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FRONT-END  ANALYSIS  METHODS  FOR  NONCOMMISSIONED  OFFICER  EDUCATION 
SYSTEM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

At  the  request  of  the  Institute  for  Noncommissioned  Officer  Professional  Development 
(INCOPD),  in  order  to  support  the  process  by  which  training  site  decisions  are  made  within  the 
Noncommissioned  Officer  Education  System  (NCOES),  this  research  effort  sought  to  identify 
the  factors  associated  with  optimal  placement  of  critical  tasks  for  training  initially  focusing  on  a 
critical  task’s  enduring  criticality  over  time.  The  primary  goal  was  to  aid  the  NCOES  in 
distinguishing  between  tasks  that  are  best  suited  for  institutional,  unit,  and  self-development 
training  sites. 

Procedure: 

Through  the  use  of  subject  matter  expert  (SME)  interviews  and  Critical  Task  and  Site 
Selection  Board  (CTSSB)  observations,  the  project  team  conducted  a  thorough  analysis  to 
identify  the  traits  that  characterize  tasks  ideally  trained  in  the  institutional  environment,  unit 
training  environment,  or  self-development  training  environment.  Using  the  outcomes  of  this 
analysis,  we  developed  a  front-end  analysis  (FEA)  methodology  based  in  these  distinguishing 
traits  that  assists  in  differentiating  those  tasks  that  are  appropriate  for  institutional  training  from 
those  tasks  that  are  better  suited  for  training  at  the  unit,  or  via  individual  self-development.  The 
FEA  methodology  supports  individuals  involved  in  recommending  training  sites  for  individual 
tasks  by  providing  a  simple  way  to  systematically  evaluate  the  characteristics  of  a  task  to 
produce  a  site  recommendation. 

The  FEA  methodology  was  instantiated  into  a  Microsoft  Excel  based  Site  Selection  Tool 
(SST).  Users  respond  to  a  series  of  questions  embodying  the  traits  identified  by  the  SME 
interviews  and  captured  in  the  FEA  methodology.  Utilizing  branching  logic  representing  the 
FEA  and  user  responses,  the  SST  makes  a  recommendation  for  training  site  location.  The 
validity  of  the  Site  Selection  Tool’s  (SSTs)  recommendations  was  assessed  via  SME  interviews 
and  three  CTSSB  pilot  tests  (using  MOSs  19A,  35T  and  35G).  The  research  team  relied  heavily 
on  user  comments  and  suggestions,  as  well  as  feedback  from  experienced  CTSSB  administrators, 
to  modify  the  tool’s  functionality  following  each  round  of  assessment.  Suggested  revisions  to 
the  interface  and  internal  logic  driving  the  SST  recommendations  were  made  based  on  universal 
suitability  to  the  Army  at  large. 

Findings: 

CTSSB  administrators,  course  managers,  and  task-analysis  groups  generally  supported 
the  incorporation  of  the  SST  into  the  CTSSB  process.  Board  members,  who  are  novice  site- 
selectors,  were  also  generally  supportive,  but  less  so  than  the  CTSSB  administrators,  as  the  board 
members  are  not  generally  aware  of  the  additional  factors  important  to  consider  in  site-selection, 
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nor  privy  to  the  pitfalls  of  a  CTSSB  conducted  relying  solely  upon  discussion.  However,  the 
SST  does  not  negate  or  overrule  human  judgment  but  instead  complements  human  judgment  in 
that  the  SST  (1)  requires  CTSSB  board  members  to  consider  important  site  selection  factors  that 
may  not  be  immediately  apparent  to  them,  and  (2)  highlights  tasks  where  high  levels  of 
disagreement  exist  in  order  to  prioritize  discussion.  Board  members  retain  the  ability  to  disagree 
with  SST’s  recommendation  and  such  disagreement  is  incorporated  into  the  SST’s  calculations 
of  consensus. 

Utilization  and  Dissemination  of  Findings: 

The  SST  is  intended  to  aid  the  Anny  at  large  in  making  better  training  site  determination 
decisions.  As  such,  it  will  be  made  available  as  widely  as  possible.  The  sponsor  for  this 
research  effort,  the  INCOPD,  will  retain  copies  of  the  SST  files  and  user  guides.  Although  the 
tool  was  developed  primarily  for  use  in  NCOES  training  decision-making,  the  tool  can  also  be 
used  for  Officer  Education  System  (OES)  and  functional  course  training  decisions.  The  tool  will 
also  be  available  for  download  on  the  Anny  Training  Network  and  through  the  U.S.  Anny 
Research  Institute.  Included  in  the  Appendices  are  user  guides  to  support  the  technical 
implementation  and  tips  and  suggestions  for  incorporating  the  SST  into  the  CTSSB  process. 
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Front-End  Analysis  Methods  for  the  Noncommissioned  Officer  Education  System 


Introduction 

At  the  broadest  level,  the  United  States  Anny  categorizes  individual  tasks  that  Soldiers 
must  perform  into  two  categories:  (1)  common  tasks,  such  as  completing  personnel  evaluations 
or  basic  rifle  marksmanship,  that  are  required  of  all  Soldiers  across  the  force  and  (2)  Military 
Occupational  Specialty  (MOS)  specific  tasks  that  are  unique  to  a  particular  branch  and  duty 
position.  For  instance,  as  MOS-specific  tasks,  an  infantryman  is  required  to  load  and  unload  the 
M240  coaxial  machine  gun  on  the  Bradley  Fighting  Vehicle,  while  an  intelligence  system 
maintainer  must  be  able  to  maintain  the  data  link  for  the  Guardrail  Common  Sensor  System.  The 
Army  further  distinguishes  between  tasks  that  are  critical  to  performance,  and  those  that  are  not. 
Critical  tasks  are  those  considered  to  be  of  utmost  importance  for  Soldiers  to  properly  perform 
their  job  as  United  States  (U.S.)  Anny  Soldiers,  and  therefore  receive  greater  attention  and 
resources  for  training.  In  addition  to  critical  tasks  that  are  common  to  all  Soldiers,  each  MOS 
maintains  a  critical  task  list  that  reflects  the  Anny  MOS  proponent’s  top  training  and  job- 
performance  priorities. 

Once  identified  as  critical,  the  responsibility  for  initially  training  the  tasks  must  be 
assigned  to  the  institution,  the  unit,  or  the  individual  Noncommissioned  Officer  (NCO)  (i.e.,  self- 
development).  Several  considerations  must  be  made  when  identifying  training  site.  The  Critical 
Task  and  Site  Selection  Board  (CTSSB)  is  a  management  device  that  provides  quality  control  for 
this  decision  process,  but  there  is  little  formalized  direction  on  how  the  optimal  training  site  is 
selected,  given  the  characteristics  of  the  tasks  or  knowledge,  skills,  and  abilities  (KSAs)  to  be 
trained.  Site  selection  factors  may  include  effectiveness  and  efficiency,  time  available  at  the 
institutions,  the  equipment  and  facilities  available,  whether  or  not  the  instructional  strategies  are 
feasible,  and  the  ability  to  engage  the  learner  with  the  media  available  (U.S.  TRADOC,  May, 
1999;  U.S.  Department  of  Defense,  June,  2004).  These  elements  have  been  employed  as  part  of 
the  recommendation  and  decision  process  for  selecting  the  appropriate  training  delivery  medium 
within  a  course  for  distance  learning,  which  is  frequently  used  as  a  training  method  for  self¬ 
development. 

While  the  unit  is  ultimately  responsible  for  ensuring  its  Noncommissioned  Officers 
(NCOs)  are  proficient  in  the  critical  tasks,  both  common  and  MOS-specific,  NCOs  also  receive 
initial  task  training  through  classroom  instruction  provided  by  the  NCO  Education  System 
(NCOES)  and  the  NCO  Academies,  and  via  self-development.  The  NCOES  is  responsible  for 
providing  institutional  training  via  the  Warrior  Leader  Course  (WLC,  focused  on  training 
Sergeants  who  are  new  NCOs),  the  Advanced  Leader  Course  (ALC,  intended  for  Staff 
Sergeants),  the  Senior  Leader  Course  (SLC,  intended  for  Sergeants  First  Class),  and  the 
Sergeants  Major  Course  (SMC,  for  new  Sergeants  Major).  Both  WLC  and  SMC  address 
common  Soldier  tasks  exclusively,  and  are  attended  by  NCOs  of  all  MOSs.  In  contrast,  each 
Army  branch/MOS’s  Noncommissioned  Officer  Academy  (NCOA)  conducts  its  own  ALC  and 
SLC,  with  a  focus  on  the  particular  skill  sets  required  of  NCOs  in  that  specific  MOS.  As  a  result, 
each  NCO  Academy  develops  and  updates  a  unique  curriculum  for  ALC  and  SLC  to  meet  the 
needs  of  its  NCO  population.  Interspersed  with  these  resident  courses  is  a  selection  of  distributed 
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Learning  (dL)  courses  that  constitute  NCO  self-development  prerequisites  for  WLC,  ALC,  SLC, 
and  SMC.  For  example,  prior  to  attending  the  MOS-specific  portion  of  ALC,  NCOs  must 
complete  the  ALC  Phase  1  dL  course,  which  is  common  core  training  for  MOS-independent 
content.  In  common  use,  dL  is  often  seen  as  synonymous  with  self-development.  Strictly 
speaking,  however,  self-development  need  not  be  conducted  via  dL,  although  dL  is  a  common 
method  employed  by  the  Army  to  provide  structure  to  self-development  tasks. 

Resourcing  and  time  constraints  seem  to  be  key  considerations  for  placing  training  in  an 
institutional  setting.  At  first  blush,  schoolhouses  may  seem  to  be  cost  effective  in  that  the 
curricula  can  be  used  repeatedly  to  train  many  Soldiers.  However,  the  transportation  and 
housing  expenses  associated  with  resident  institutional  courses  may  offset  gains  associated  with 
mass  delivery  of  training.  In  line  with  this  reasoning,  following  an  extensive  review  by  the 
Defense  Science  Board  Task  Force  prior  to  the  events  of  September  1 1 ,  200 1 ,  the  Office  of  the 
Undersecretary  of  Defense  for  Acquisition  recommended  that  a  full  half  of  schoolhouse  training 
should  be  moved  to  unit-based  training  within  the  upcoming  five  years  (Braddock  &  Chatham, 
2001).  Such  a  move  would  have  saved  the  Anny  at  least  $  1 14,000,000  per  year  in  per  diem 
costs  alone.  More  savings  would  have  been  realized  in  terms  of  lessening  unit  disruption.  The 
events  of  September  1 1th  precluded  this  from  occurring,  yet  the  goal  of  moving  much  of 
institutional  training  to  the  unit  remains  today.  Moreover,  as  the  implementation  of  the  recent 
Anny  Learning  Model  (ALM;  U.S.  TRADOC,  January,  2011a)  takes  root  and  gains  traction,  a 
greater  onus  will  be  placed  on  NCOs  to  be  accountable  for  their  own  learning  and  become  active 
participants  in  their  own  training.  This  is  true  not  only  in  fonnal  training  environments,  but  also 
outside  of  the  classroom  as  reflected  by  the  Army’s  growing  emphasis  on  encouraging  NCOs  to 
be  lifelong  learners.  Ideally,  an  underlying  match  would  exist  between  task  and  training  site  that 
targets  the  core  nature  of  the  task  rather  than  resource  and  logistical  concerns. 

The  rationale  for  distinguishing  tasks  that  are  enduring  across  operational  environments 
from  tasks  that  are  specific  to  a  current  operational  environment  stems  in  part  from  the  time  and 
resource  demands  associated  with  revising  institutional  programs  of  instruction  (POI).  Because 
POI  changes  are  labor  intensive  and  time  consuming,  the  institutional  training  environment  is  not 
particularly  conducive  to  rapid  modifications  to  curricula  as  a  result  of  current  operational 
demands.  To  support  recent  combat  operations  in  Iraq  and  Afghanistan,  however,  institutions 
have  in  fact  diverted  their  time  and  resources,  and  adapted  their  curricula,  to  meet  the  new 
training  requirements  that  have  emerged  as  a  result  of  evolving  mission  requirements.  This 
approach  has  arguably  produced  an  NCO  corps  better  equipped  for  combat  operations.  Yet 
concurrently,  it  has  recast  the  institution  as  a  mechanism  to  broadly  deliver  new  and  high-priority 
training,  leveraging  the  institution’s  reach  to  huge  populations  of  NCOs.  For  instance,  the  WLC, 
which  is  attended  by  infantrymen,  cooks,  and  veterinarians  alike,  added  counter-improvised 
Explosive  Device  (CIED)  training  to  the  course  to  ensure  all  NCOs  receive  at  least  rudimentary 
skills  to  avoid  what  is  currently  the  most  significant  battlefield  threat.  Similarly,  institutional 
training  has  also  been  leveraged  to  deliver  refresher  training  on  basic  Army  skills  that  may  not  be 
the  primary  purpose  of  a  course.  Often,  it  is  unclear  which  tasks  and  training  requirements 
reflect  new  and  enduring  tasks  that  warrant  institutional  training,  vice  new  but  temporary 
requirements  that  may  not  be  optimally  suited  for  institutional  training.  As  the  Anny  transitions 
away  from  the  heavy  operational  tempo  of  the  past  decade  of  war,  and  implements  the  tenets  of 
the  ALM,  the  Army  can  realign  its  training  goals  to  reflect  ideal  training  environments. 
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Aside  from  resource  and  logistical  concerns,  however,  there  remains  a  more  fundamental 
concern  regarding  whether  some  tasks  are  differentially  suited  for  training  in  the  institution,  at 
the  unit,  or  via  self-development.  Task  characteristics  and  training  demands  may  more 
appropriately  dictate  the  optimal  placement  of  a  task  in  a  training  site.  Traditionally,  Army 
institutional  training  has  been  reserved  for  addressing  enduring  individual  skills  that  represent 
the  core  skill  sets  of  NCOs,  and  emphasizing  the  doctrinally  correct  approach.  In  contrast,  the 
unit  trains  NCOs  for  the  particular  knowledge  or  task  requirements,  and  tactics,  techniques,  and 
procedures  (TTPs),  specific  to  its  current  or  upcoming  mission.  For  example,  a  medical  unit 
expecting  to  encounter  a  predominance  of  a  certain  type  of  injury  on  its  next  deployment  may 
well  focus  unit  training  time  on  critical  tasks  that  reflect  those  likely  injuries.  Similarly,  a  unit 
deploying  to  Iraq  conducts  different  language  and  culture  preparation  than  one  deploying  to 
Afghanistan.  In  short,  the  institution’s  traditional  role  is  to  train  doctrinally  correct  skills 
required  by  all  NCOs  within  an  MOS  or  across  the  force,  while  unit  training  typically  addresses 
unit  Standard  Operating  Procedures  (SOPs)  and  TTPs  that  reflect  current  needs  and  best 
practices  from  theater.  From  a  time  and  resource  perspective,  regularly  updating  institutional 
POIs  is  costly.  To  make  matters  worse,  the  time-consuming  and  lengthy  nature  of  modifying 
POIs  may  result  in  initially  highly  important  tasks  becoming  out-of-date  by  the  time  that  the  POI 
modification  is  approved  and  implemented.  Thus,  some  tasks  continue  to  be  trained  in  an 
institutional  environment  past  the  time  when  they  were  deemed  critical.  The  Army’s  process  for 
updating  the  assignment  of  tasks  to  institutional,  unit,  or  self-development  training  sites,  occurs 
roughly  once  every  three  years  via  the  CTSSB  procedure  (TRADOC,  2004).  As  a  result,  tasks 
may  linger  in  an  institutional  POI  long  past  their  prime,  resulting  in  valuable  institutional 
training  time  spent  in  non-optimal  ways.  As  institutional  training  courses  are  governed  by  strict 
schedules,  time  spent  training  these  low  priority  or  even  unnecessary  tasks  could  be  better 
allocated  to  more  important  tasks.  Thus,  task  endurability  has  come  into  question  as  a  primary 
consideration  for  site  selection. 

Adaptations  of  the  institution’s  training  mission  have  resulted  in  changes  to  its  traditional 
role  and  indications  from  cadre  and  students  that  task  training  sites  are  misaligned.  Students  and 
cadre  have  expressed  concern  that  valuable  institutional  training  time  is  spent  training  content 
specific  to  current  operations  that  is  unrelated  to  a  course’s  primary  mission  and  is  outside  the 
proponent’s  domain.  Another  concern  is  that  institutions  train  material  that  NCOs  have  already 
learned  on  the  job.  This  muddying  of  the  traditional  distinctions  between  training  site  purpose 
has  led  to  redundancy,  such  as  when  training  content  is  repeated,  at  times  identically,  across 
multiple  training  sites.  Conversely,  there  has  been  an  inappropriate  elimination  of  some  critical 
tasks  from  institutional  training  due  to  time  and  resource  constraints,  and  a  resistance  to 
increasing  the  length  of  the  course  to  accommodate  for  added  training  content. 

The  objective  of  this  research  effort  was  to  identify  the  factors  associated  with  optimal 
placement  of  tasks  for  training,  in  order  to  support  the  process  by  which  such  decisions  are  made 
within  the  NCOES.  The  research  was  framed  with  the  assumption  of  endurability  as  a  primary 
discriminating  factor,  but  left  open  the  potential  for  other  factors  or  variables  to  discriminate 
between  training  site  suitability.  While  we  recognize  the  value  of  the  research  outcomes  for 
better  understanding  training  site  selection  factors,  the  primary  goal  was  to  aid  the  NCOES  in 
distinguishing  between  tasks  that  are  best  suited  for  institutional,  unit,  and  self-development 
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training  sites.  To  that  end,  we  conducted  a  thorough  analysis  of  the  traits  that  characterize  a  task 
ideally  trained  in  the  institutional  environment,  traits  of  a  task  ideal  for  unit  training,  and  those  of 
a  task  ideally  suited  to  self-development.  Using  the  outcomes  of  this  analysis,  we  developed  a 
front-end  analysis  (FEA)  methodology  that  assists  in  differentiating  those  tasks  that  are 
appropriate  for  institutional  training  from  those  tasks  that  are  better  suited  for  training  at  the  unit 
level,  or  via  individual  self-development.  The  FEA  methodology  would  therefore  support 
individuals  involved  in  recommending  training  sites  for  individual  tasks,  by  providing  a  simple 
way  to  systematically  evaluate  the  characteristics  of  a  task  to  produce  a  site  recommendation. 

Our  approach  to  this  research  effort  included  a  literature  review  to  identify  the  Anny’s 
current  processes  related  to  placing  tasks  for  training,  and  other  service  and  industry  practices 
associated  with  matching  tasks  and  training  requirements  to  sites  and  media.  The  Background 
section  of  this  report  describes  these  practices  and  associated  challenges  to  more  clearly  frame 
the  research  requirement  and  the  context.  Upon  completion  of  the  literature  review,  an  analysis 
of  site  selection  factors  commenced,  as  described  in  the  Analysis  of  Site  Selection  Factors 
section,  to  identify  the  most  critical  task  characteristics  for  consideration  when  identifying  the 
optimal  site  for  training.  The  focus  of  this  research  effort  was  the  NCOES,  which  encompasses 
WLC,  ALC,  and  SLC.  SMC  was  not  included  in  this  study.  As  such,  the  researchers  conducted 
observations  of  the  Army’s  CTSSB  process  in  which  the  training  site  determination 
recommendations  are  made,  and  interviewed  board  members  and  subject  matter  experts  (SMEs) 
with  knowledge  and  experience  in  ascertaining  the  qualities  of  tasks  that  are  most  amenable  to 
institutional,  unit,  and  self-development  training.  From  these  observations  and  interviews,  an 
analysis  was  conducted  that  revealed  several  consistent  factors  indicating  variables  upon  which 
tasks  could  be  discriminated  for  the  purposes  of  determining  training  site  location.  These  factors 
were  then  formulated  into  a  logic  tree  reflecting  the  priority  and  order  of  consideration  reflecting 
SMEs’  distinctions  between  training  site  considerations.  The  resulting  logic  tree  allowed  for  the 
generation  of  the  FEA  methodology  for  recommending  placement  of  critical  tasks  into  a  training 
site.  This  methodology  was  then  embedded  into  a  Site  Selection  Tool  (SST)  to  support  novice 
site  selectors  and  improve  the  quality  of  their  recommendations.  A  popular,  user-friendly, 
widely-available  and  U.S.  Department  of  Defense  (DoD)  supported  software  was  utilized  to 
maximize  ease  of  adoption.  The  procedure  and  outcomes  of  the  FEA  methodology  and  SST 
development  efforts  are  detailed  in  the  Site  Selection  Methodology  and  Tool  section.  Next,  the 
SST  was  assessed  via  multiple  iterations  in  Anny  CTSSBs  with  the  Annor  School  and  the 
Military  Intelligence  School.  After  each  assessment  and  implementation  of  the  tool,  the  research 
team  considered  and  included  a  selection  of  proposed  revisions  based  on  user  feedback.  The 
assessment  process  and  key  outcomes  influencing  revisions  to  the  SST  are  described  in  the 
section  titled  Site  Selection  Tool  Assessment.  Finally,  program  outcomes  and  implications  for 
use  of  the  products  are  presented  in  the  Discussion  and  Conclusions. 


Background 

In  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Pamphlet  350-70-6,  the 
Anny  outlines  its  process  for  reviewing  and  updating  critical  task  lists.  This  process  provides 
guidelines  for  selecting  critical  tasks  for  each  MOS  as  part  of  the  CTSSB  process.  CTSSBs  are 
responsible  for  refining  an  MOS  critical  task  list  and  recommending  training  sites.  CTSSBs  are 
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conducted  for  all  NCOES  courses  by  each  course/MOS  proponent  and  result  in  an  updated 
critical  task  list  for  the  MOS. 

In  line  with  the  Army’s  analysis,  design,  development,  implementation,  and  evaluation 
(ADDIE)  process  for  training,  TRADOC  guidelines  indicate  that  this  review  process  should 
occur  every  three  years  and  include  a  comprehensive  review  of  the  tasks  that  are  conducted  by 
NCOs  in  a  given  MOS.  While  TRADOC  presents  guidelines  for  both  individual  and  collective 
tasks  in  TRADOC  Pamphlet  350-70-6,  this  research  effort  focused  solely  on  individual  tasks. 
See  Figure  1  for  an  overview  of  this  process,  as  described  in  Pamphlet  350-70-6. 


Figure  1.  Total  task  inventory1 


Figure  1  provides  an  overview  of  the  prescribed  process,  but  in  practice,  some  departures 
occur.  In  brief,  the  process  includes  the  following  steps. 

CTSSBs  are  conducted  on  a  three  year  schedule,  or  sooner  if  a  need  arises  based  on 
critical  incidents,  performance  deficiencies,  law  or  doctrine  changes,  or  for  other  reasons.  Upon 
instigation,  the  first  phase  is  a  job  analysis,  generally  conducted  by  training  developers  (General 
Schedule  1750)  and  SMEs  in  the  content  domain.  The  job  analysis  phase  involves  soliciting 
feedback  from  individuals  holding  the  duty  position  under  review.  To  begin  the  job  analysis, 
first  a  Total  Task  Inventory  list  must  be  developed.  This  list  represents  the  totality  of  tasks  that 
the  organizers  (training  developers,  task  analysis  group,  etc.)  believe  the  CTSSB  should  consider 
for  a  critical  task  list.  The  list  is  compiled  from  various  resources  including  existing  critical 
tasks  lists,  subject  matter  experts,  collective  task  lists,  mission  analysis,  and  analysis  of  surveys 
and  other  task  perfonnance  data.  See  Figure  2  below. 


1  Originally  published  in  TRADOC  PAM  350-70-6  (2004). 
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A  comprehensive  Total 
Task  Inventory  is  compiled 
from  many  sources. 


Figure  2.  Building  a  total  task  inventory” 


The  process  relies  heavily  upon  input  from  SMEs,  those  individuals  who  possess  a 
thorough  knowledge  of  the  job,  including  the  duties  of  the  job  as  well  as  the  tasks.  Because  SME 
input  is  essential  to  developing  the  Total  Task  Inventory,  it  is  important  that  only  those  SMEs 
with  the  highest  level  of  expertise  are  selected  for  input  into  this  process  (U.S.  TRADOC, 
September,  2004).  Once  the  Total  Task  Inventory  is  built,  a  job  analysis  survey  is  administered 
to  individuals  holding  the  MOS.  This  data  can  be  collected  by  automated  systems,  such  as 
AUTOGEN.2 3 


The  purpose  of  the  job  analysis  survey  is  to  gamer  infonnation  about  the  role  of  the  tasks 
in  job  performance  for  the  MOS  under  review.  Training  developers  base  the  specific 
infonnation  they  solicit  about  these  tasks  on  task  selection  models,  of  which  a  variety  are 
available  across  the  Anny.  Their  main  function  is  to  prioritize  and  rank  the  tasks  in  order  of 
their  importance  in  several  different  areas. 

Eleven  task  selection  models  are  recognized  doctrinally,  though  the  following  five 
models  are  the  ones  most  frequently  used  to  judge  task  criticality: 


2  Originally  published  in  TRADOC  PAM  350-70-6  (2004). 

3  AUTOGEN  is  an  ARI  survey  product  used  to  collect  data.  Beginning  in  FY13,  this  product  will  not  be  available 
thru  ARI,  including  training  developers  or  others  who  wish  to  utilize  it  for  their  training  analysis  needs  as  part  of  the 
CTTSB  process. 
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•  Difficulty-Importance-Frequency  Model  -  Tasks  are  identified  as  critical  based  on  the 
difficulty,  importance,  and  frequency  with  which  they  are  perfonned. 

•  Four  Factor  Model  -  Tasks  are  identified  as  critical  based  on  their  percent  of  total  job 
perfonnance  and  their  difficulty  to  leam. 

•  Probability  of  Task  Criticality  Model  -  Tasks  are  identified  as  critical  based  on 
occupational  data,  analysis,  requirements,  and  structure  of  the  program. 

•  Training  Emphasis  (TE)  Model  -  Tasks  are  identified  as  critical  based  on  the  TE 
factor,  which  is  collected  from  supervisors  and/or  jobholders.  It  reflects  the  job 
incumbent’s  and  the  supervisor’s  judgment  concerning  how  much  emphasis  to  give 
the  task  in  training.  The  TE  is  the  most  used  single  training  factor  for  critical  task 
selection. 

•  Eight  Factor  Model  -  Tasks  are  identified  as  critical  based  on  judges’  ratings  of  the 
tasks  on  a  scale  from  one  (1)  to  six  (6),  with  a  higher  rating  indicating  greater 
importance,  on  eight  separate  dimensions.  That  is,  tasks  are  identified  as  critical 
based  on  (1)  the  percent  of  NCOs  perfonning  the  task  (across  the  Army),  (2)  the 
percent  of  time  spent  performing  this  task,  (3)  the  severity  of  the  consequence  of 
inadequately  perfonning  the  task,  (4)  the  tolerance  of  the  task  to  a  delay  in 
conducting  it,  (5)  the  frequency  of  performing  this  task,  (6)  the  difficulty  in  learning 
this  task,  (7)  the  probability  of  deficient  performance,  and  (8)  the  immediacy  of 
perfonning  the  task  (i.e.,  how  automated  perfonnance  must  become). 

The  manner  in  which  the  job  analysis  is  distributed  varies  across  proponents  from  a  top- 
down  directive  to  participate  to  word-of-mouth  and  from  specific  requests  for  sponsoring  units  to 
mass-email  solicitations.  The  results  of  the  job  analysis  survey  are  analyzed  and  prepared  for 
presentation  to  the  CTSSB  members  to  guide  their  decisions  regarding  task  criticality.  Thus,  in 
addition  to  the  subjective  nature  of  ratings,  these  models  are  used  to  “apply  statistically  valid  task 
selection  data  to  identify  critical  individual  tasks”  (p.  81,  U.S.  TRADOC,  September,  2004). 
Although  the  job  analysis  is  intended  to  precede  the  board’s  convention,  this  step  is  sometimes 
replaced  by  board  members  providing  their  own  ratings  of  the  tasks  in  the  total  task  inventory 
during  the  CTSSB,  or  sometimes  may  be  dropped  entirely  if  time  and  resources  do  not  permit,  in 
favor  of  a  simple  critical/non-critical  vote. 

The  development  of  the  total  task  inventory  and  the  job  analysis  are  to  be  conducted  prior 
to  the  board’s  convening.  For  the  actual  board  meeting,  representatives  from  the  U.S.  Army 
Forces  Command  (FORSCOM)  (including  Reserve  Component)  are  solicited.  The  manner  in 
which  they  are  solicited  varies,  but  the  goal  is  to  secure  representation  by  individuals  with  recent 
and  valuable  operational  experience  from  a  wide  range  of  backgrounds.  At  times,  individuals  are 
hand-selected  or  personally  invited  by  proponents,  while  other  times  individuals  are  nominated 
at  units’  discretion.  Voting  members  should  be  supervisors  of  (one  or  two  levels  above)  the  duty 
position  under  review  and  must  include  representation  from  National  Guard  and/or  Army 
Reserve.  Boards  may  also  include  non-voting  members  with  subject  matter  expertise  to  include 
SMEs  of  higher  or  lower  rank,  civilian  training  developers,  instructors,  and  so  forth.  Boards  are 
facilitated  by  a  non-voting  chairperson. 

CTSSBs  are  generally  a  week  in  duration  and  typically  consist  of  four  primary  objectives, 
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each  of  which  maps  to  a  set  of  activities  within  the  CTSSB: 


1.  Introduction,  instructions,  and  guidance.  The  CTSSB  begins  with  the  administrator 
and  chairperson  setting  the  stage  for  the  board  members.  During  this  time,  board 
members  are  briefed  on  the  purpose  and  outcomes  of  the  CTSSB.  They  are  oriented 
on  how  to  apply  their  experience  to  the  goals  of  the  CTSSB.  And,  they  are  given 
instructions  for  how  to  rate  task  criticality  and  recommend  training  sites  for  each  task. 

2.  Updating  the  task  list.  The  next  stage  of  the  CTSSB  usually  consists  of  reviewing  the 
task  list  in  question,  making  a  determination  as  a  group  as  to  which  tasks  should  be 
deleted  or  modified,  and  suggesting  tasks  to  be  added  to  the  list.  In  other  words,  the 
board  revises  the  MOS  task  list  to  ensure  it  is  keeping  step  with  requirements  for 
NCOs  as  they  support  their  units.  As  previously  discussed,  the  one  exception  to  this 
activity  lies  at  the  WLC  CTSSB,  which  is  run  by  the  U.S.  Anny  Sergeants  Major 
Academy  (USASMA).  The  common  core  tasks  reviewed  there  are  maintained  by  a 
range  of  MOSs  across  the  Anny,  and  are  therefore  not  within  the  purview  of 
USASMA  to  delete  or  revise. 

3.  Ratings  of  task  criticality.  In  the  third  stage  of  the  CTSSB,  the  board  applies  one  of 
the  task  selection  models  previously  discussed  to  rate  the  criticality  of  each  task  on 
the  list.  At  the  end  of  this  stage,  every  task  has  a  criticality  rating  associated  with  it, 
usually  on  a  scale  from  1  to  5  or  1  to  6.  In  addition,  the  board  administrators  typically 
identify  a  criticality  cut  off  point  (e.g.,  3.2  on  a  5-point  scale),  where  all  tasks  falling 
at  or  above  that  rating  are  deemed  critical  and  thus  become  higher  priority  training 
requirements. 

4.  Site  selection.  The  final  stage  of  the  CTSSB  is  site  selection.  At  the  discretion  of  the 
board  administrator,  the  board  may  recommend  initial  training  sites  for  only  the 
critical  tasks,  or  for  all  tasks  on  the  list.  As  we  have  described,  the  board  may 
recommend  institution,  unit,  or  self-development.  Within  the  institution  option,  a 
board  may  identify  a  functional  course  rather  than  an  NCOES  course  to  address 
training  for  the  task. 

The  board  begins  its  process  of  updating  the  task  list  by  reviewing  the  Total  Task 
Inventory  list  developed  by  the  job/task  analysis  or  training  development  team.  If  a  job  analysis 
survey  was  conducted,  the  results  may  be  presented  to  the  board.  Either  on  the  basis  of  the  job 
analysis  results  and/or  in  conjunction  with  board  member  expertise,  the  board  votes  to  rate  task 
criticality  of  each  task  in  the  total  task  inventory.  Specific  voting  procedures  vary,  but  may 
include  numerical  ratings  (e.g.,  a  1-5  scale  with  a  specified  cut-point),  or  a  simple  yes/no  vote. 
Specific  frameworks  also  vary,  as  does  adherence  to  a  single  framework;  however,  one  of  the 
aforementioned  task  selection  models  is  frequently  applied.  The  voting  procedure  usually 
includes  discussion  on  each  task.  As  would  be  expected,  group  dynamics  and  individual 
personalities  can  at  times  have  strong  influences  on  the  votes  of  individual  board  members.  As 
might  be  expected,  board  results  tend  to  reflect  the  specific  backgrounds  of  the  individuals  who 
comprise  the  board,  reflecting  unique  perspectives  of  their  own  units’  missions  and  experiences 
regarding  what  constitutes  a  critical  task  and  what  does  not.  Detennination  of  what  qualifies  as  a 
critical  task  varies,  and  no  firm  rules  guide  this  decision.  Usually,  the  score  qualifying  for 
criticality  is  set  either  by  precedent  or  by  the  board  chair  or  board  administrator.  In  the  case  of  a 
numerical  rating,  each  task  is  generally  assigned  a  score  taken  as  a  calculated  average  of  each 
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board  member’s  rating;  if  the  average  score  is  above  the  designated  critical  value  (e.g.,  a  3  or 
higher  on  a  scale  of  1-5),  it  becomes  a  critical  task.  In  the  case  of  a  simple  yes/no  vote  for 
criticality,  the  board  chair  or  administrator  will  set  an  acceptable  consensus  level  (frequently  a 
simple  majority).  In  conjunction  with  establishing  a  critical  task  list,  boards  may  take  the 
opportunity  to  modify  the  tasks  by  revising  portions  of  a  task  title,  description,  and  at  times, 
conditions  and  standards.  Tasks  may  also  be  added  to  the  list  as  recommended  new  tasks  for  the 


MOS. 


Following  the  identification  of  critical  tasks,  all  boards  include  a  site  selection  phase. 
While  several  models  to  establish  task  criticality  can  be  identified,  no  fonnal  or  standardized 
mechanism  exists  for  initial  training  site  determinations,  either  within  the  CTSSB  context  or 
elsewhere  in  the  Anny  systems.  However,  the  Army  has  begun  to  focus  more  attention  on 
training  site  detennination.  An  integrated  training  environment  (ITE),  which  incorporates  both 
institutional  and  unit-based  training,  is  currently  underway  (U.S.  TRADOC,  January,  2011b). 
This  holistic  ITE  will  synchronize  and  balance  both  unit  and  institutional  training  environments. 
This  approach  recommends  that  NCOs  and  leaders  should  be  trained  to  develop  their  individual 
cognitive,  interpersonal,  and  cultural  skills  in  the  institutional  learning  environment.  These 
individual  skills  are  then  transfonned  from  individual,  adaptive  skills  into  collective,  adaptive 
skills  to  meet  unit  readiness,  via  unit  training  that  is  better  equipped  to  replicate  the  difficulties 
and  complications  of  the  current  operational  environment.  By  introducing  complex  and 
challenging  unit  training  opportunities,  leaders  and  units  gain  the  experience  and  skills  required 
to  execute  full-spectrum  operations.  As  such,  the  ITE  provides  commanders  and  leaders  with  the 
flexibility  and  adaptability  in  planning,  preparing,  and  executing  both  types  of  training  programs, 
whether  institutionally  based  or  unit-based  (U.S.  TRADOC,  January,  2011b). 

Within  the  CTSSB  context,  the  site  detennination  process  is  less  standardized  and  is 
given  less  emphasis  than  the  task  criticality  phase.  While  members  of  the  CTSSB  bring 
substantial  operational  experience  and  expertise  to  the  board,  seldom  does  a  board  member  have 
any  prior  experience  sitting  on  a  CTSSB  or  selecting  sites  for  task  training.  In  other  words,  all 
board  members  are  novice  site  selectors,  unaided  by  any  fonnal  process  or  guidance.  While  their 
MOS-related  expertise  and  their  duty  position  experience  is  highly  valued  and  gives  them  the 
necessary  knowledge  to  properly  characterize  a  task,  board  members  are  unlikely  to  know  which 
characteristics  about  a  task  align  it  best  with  a  specific  training  site. 

The  site  selection  phase  of  the  CTSSB  generally  includes  an  informal  discussion,  and 
usually,  but  not  always,  a  board  vote.  Often,  one  or  two  board  members  make  a  general 
recommendation  for  training  site,  and  this  recommendation  prevails  unless  strong  disagreement 
is  voiced.  Informally,  it  has  been  suggested  that  a  bias  exists  toward  assigning  tasks  to  an 
institutional  setting,  for  three  possible  reasons.  First,  there  seems  to  be  an  implicit  belief  that  the 
institution  trains  the  most  critical  individual  tasks,  while  the  unit  focuses  on  training  collective 
tasks  and  those  individual  tasks  of  lesser  criticality.  Secondly,  the  board  is  intentionally 
comprised  of  individuals  from  units.  Recommending  that  a  task  be  trained  at  the  unit  (such  as 
their  own)  essentially  increases  the  board  members’  requirement  to  find  training  time  and  access 
facilities  and  equipment  at  their  home  unit,  so  there  may  be  reluctance  toward  assigning  tasks  to 
unit  training.  Finally,  site  selection  recommendations  are  arguably  influenced  by  current 
practices,  meaning  that  tasks  currently  trained  at  the  institution  may  continue  to  be  recommended 
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for  institutional  training  due  to  board  member  familiarity  with  such  placement.  The  outcomes  of 
this  research  effort  were  thus  intended  to  lend  a  degree  of  standardization  to  the  site  selection 
process,  improve  its  effectiveness  and  efficiency,  and  streamline  the  decision  process  to  give 
board  members  a  tool  with  which  to  make  training  site  recommendations  that  are  grounded  in 
criteria  established  by  SMEs  in  site  detennination. 

Following  the  CTSSB,  course  personnel  review  the  CTSSB’s  recommendations  for  the 
critical  task  list  and  site  selection,  apply  their  own  knowledge  and  experience  regarding 
appropriate  site  placement,  and  submit  their  recommendations  to  the  Commandant.  The 
Commandant  makes  any  edits  he/she  deems  necessary,  accounting  for  his/her  own  expertise  and 
understanding  of  Anny  needs  and  available  resources.  Finally,  the  new  critical  task  list  is 
published,  and  in  the  case  of  institutional  and  self-development  training,  course  curricula  are 
developed  to  account  for  any  newly  included  tasks. 

There  are  two  noteworthy  exceptions  to  this  general  process:  the  Warrior  Leader  Course 
(WLC)  and  the  Sergeants  Major  Course  (SMC).  WLC  provides  training  on  common  core  tasks 
for  SPCs  and  SGTs  across  the  Anny  and  SMC,  provides  training  on  common  core  tasks  for  all 
Sergeants  Major.  The  USASMA  is  responsible  for  the  content  of  these  two  courses  and  updates 
to  the  curriculum,  but  does  not  have  the  authority  to  modify  the  tasks  themselves,  which  are 
created  and  maintained  by  other  proponents.  The  USASMA  does  not  have  the  authority  to  edit 
tasks  belonging  to  other  proponents,  so  the  board  can  only  vote  in  regards  to  whether  the  task,  as 
it  is  written,  should  be  critical  for  the  respective  course.  As  the  CTSSBs  for  these  two  courses 
focus  on  preparing  NCOs  on  common  tasks  necessary  for  NCOs  at  the  beginning  (i.e.,  WLC) 
and  culmination  (i.e.,  SMC)  of  their  service  in  the  NCO  corps,  the  tasks  that  these  boards 
consider  for  inclusion  in  their  respective  courses  are  much  more  varied  and  represent  tasks  from 
across  many  Army  MOSs  and  duty  positions,  such  as  basic  infantry  soldiering  skills  (e.g.,  land 
navigation),  general  leadership  (e.g.,  completing  NCO  Evaluation  Reports),  and  professional 
development  lessons  (e.g.,  sexual  harassment  training).  As  such,  these  boards  are  comprised  of 
individuals  across  multiple  MOSs.  Within  single  MOS  CTSSBs,  opinion  and  expertise 
variations  exist  and  can  spur  heated  discussion.  This  effect  can  be  even  more  pronounced  for 
CTSSBs  that  must  broach  a  considerably  broader  and  more  diverse  array  of  Anny  experiences. 

Analysis  of  Site  Selection  Factors 

The  requirement  for  an  improved  capability  to  detennine  optimal  training  sites  is 
apparent.  To  fill  the  void  with  guidance  and  support  for  the  site  selection  recommendation 
process,  researchers  conducted  an  analysis  of  factors  in  support  of  site  selection  decisions  or 
recommendations  by  skilled  and  experienced  individuals.  The  site  selection  factors  were 
identified  through  data  collection  in  the  fonn  of  SME  interviews  and  CTSSB  observations,  as 
well  as  a  qualitative  data  analysis.  Characteristics  associated  with  enduring  and  non-enduring 
tasks  were  also  investigated.  The  analyses  described  herein  fonned  the  basis  of  the  SST  design, 
to  improve  the  effectiveness  and  efficiency  of  site  selection  recommendations  by  CTSSBs.  The 
following  sections  detail  first  the  identification  of  critical  factors,  the  development  of  the  FEA 
methodology  and  logic,  and  then  the  technical  development  of  the  SST  which  conveys  the  FEA 
in  the  fonn  of  a  user-interactive  tool.  The  assessment  and  revisions  of  the  SST  are  then 
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presented  to  explain  how  the  SST  incorporated  user  feedback  and  additional  modifications  to 
improve  functionality. 


Methodology 


Determining  Site  Selection  Factors 

Participants.  Participants  in  the  SME  interviews  were  obtained  telephonically  and  from 
three  site  visits:  Fort  Bliss,  TX,  Fort  Benning,  GA,  and  Fort  Huachuca,  AZ.4  Interviewee 
criteria  were  set  forth  as  follows: 

•  Individual  possesses  exceptional  knowledge  and  experience  in  identifying  tasks  that 
are  best  suited  for  institutional  training,  unit  training,  or  individual  self-development. 

•  Individual  is  experienced  in  identifying  tasks  that  were  misplaced  in  institutional 
training  when  they  were  better  suited  for  unit  training,  or  vice  versa. 

•  Individual  is  experienced  in  CTSSBs. 

•  Individual  is  familiar  with  the  Army  Learning  Model  (ALM)  and  its  implications  for 
developing  and  placing  training. 

Most  interview  candidates  were  identified  in  advance  of  the  research  team’s  visit  to  each 
site,  with  attempts  to  match  the  criteria  as  closely  as  possible.  These  interviewees  represented 
job  roles  such  as  Noncommissioned  Officer  Education  System  (NCOES)  course  chiefs  and 
course  managers;  Noncommissioned  Officer  Academy  (NCOA)  former  and  current 
commandants  and  deputies;  training  and  course  developers;  NCOES  senior  instructors;  and 
other  key  individuals  in  the  Task  Analysis  and  Instructional  Development  branches.  All 
interviewees  were  either  active  duty  personnel,  or  retired  Army  officers  or  NCOs  currently 
serving  as  government  civilians.  Additional  interviewees  were  selected  once  on  site,  and  these 
individuals  were  all  first-time  board  members  (active  duty)  representing  the  operational  force  on 
a  CTSSB.  As  such,  they  were  not  SMEs  in  site  selection  per  se,  but  represented  a  user 
population  whose  perspective  required  consideration.  Finally,  telephonic  interviews  were 
conducted  with  representatives  from  the  Coast  Guard’s  Perfonnance  Technology  Center.  In  all, 
35  individuals  were  interviewed  in  32  interview  sessions.  Table  1  summarizes  the  interview 
participants  by  site. 

Table  1 


Summary  of  Interview  Participants  by  Site  and  Type 


Interview  Site 

Number  of 
Active  Duty 
SMEs 

Number  of 
Civilian  SMEs 

Number  of 
CTSSB  Members 
(non-SMEs) 

Totals 

Fort  Bliss 

7 

3 

4 

14 

Fort  Benning 

7 

4 

1 

12 

Fort  Huachuca 

1 

5 

6 

Telephonic  (Coast  Guard) 

1 

2 

3 

Totals 

16 

14 

5 

35 

4  Individuals  interviewed  at  the  three  sites  were  not  necessarily  residents  of  the  sites.  A  subset  of  the  participants 
were  on  temporary  duty  when  interviewed. 
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Observations  were  conducted  at  two  sites.  Researchers  observed  portions  of  a  Fort  Bliss 
CTSSB,  which  reviewed  common  core  tasks  for  criticality  and  inclusion  in  Warrior  Leader 
Course  (WLC),  and  a  Fort  Benning  Infantry  Officer  (1 1A)  CTSSB,  which  reviewed  infantry 
Lieutenant  (LT)  and  Captain  (CPT)  tasks  for  inclusion  in  the  Infantry  Basic  Officer  Leader 
Course  and  the  Maneuver  Captain’s  Career  Course,  respectively.  Although  the  latter  was  not  an 
NCOES  board,  we  relied  on  observations  of  the  board’s  process,  which  are  sufficiently  similar  to 
NCO  boards  to  warrant  inclusion  in  the  data  collection.3 

Materials.  Interviews  were  conducted  using  an  Informed  Consent  Form,  a  Privacy  Act 
Statement,  Task  Lists,  and  a  Data  Collection  Protocol. 

Informed  Consent  Form.  The  Infonned  Consent  Form  (see  Appendix  A)  introduced  the 
purpose  of  the  study,  described  the  interview  objectives,  clearly  indicated  that  participation  was 
voluntary  and  could  be  tenninated  at  any  time,  and  assured  interviewees  that  their  responses 
would  not  be  attributable  to  them  personally.  In  addition,  points  of  contact  were  provided  for 
questions  concerning  the  study  and  questions  about  individuals’  rights  as  interviewees.  The 
participant  signed  the  consent  form  to  acknowledge  that  the  he/she  was  volunteering  to 
participate,  was  at  least  1 8  years  of  age,  and  was  willing  to  have  his  or  her  interview  digitally 
recorded.5 6  Interviewees  retained  the  Consent  Form  for  their  records,  while  the  researchers 
retained  the  signature  pages. 

Privacy  Act  Statement.  Similar  to  the  Infonned  Consent  Form,  the  Privacy  Act 
Statement  (see  Appendix  B)  summarized  the  purpose  of  the  study  and  reminded  interviewees  of 
their  rights  to  confidentiality  and  voluntary  participation.  Interviewees  retained  a  copy  of  the 
Privacy  Act  Statement  for  their  records. 

Task  Lists.  Prior  to  data  collection  at  each  site,  the  task  list  to  be  reviewed  at  the  CTSSB, 
or  in  the  case  of  Fort  Huachuca,  relevant  to  the  interviewees,  was  obtained.  Lists  were  culled  for 
tasks  whose  training  placement  appeared  to  the  research  team’s  SME  to  fall  into  one  of  the 
following  categories:  clearly  institution  trained;  clearly  unit  trained;  clearly  appropriate  for  self¬ 
development;  new  (i.e.,  recently  added)  tasks;  and  controversial  tasks  and  tasks  whose  placement 
is  not  apparent.  Once  tasks  were  identified  that  were  believed  to  fall  into  those  categories,  two 
or  three  partial  lists  of  60-80  tasks  were  compiled  for  use  as  examples  to  be  discussed  during  the 
interviews. 


5  The  Infantry  and  Armor  NCO  and  Officer  boards  are  conducted  by  the  same  office,  and  therefore  utilize  nearly 
identical  procedures  across  the  NCO  and  Officer  boards. 

6  Digital  recordings  were  made  solely  to  facilitate  note-taking.  The  participants  were  made  aware  of  this  purpose. 
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Data  Collection  Protocol.  The  data  collection  protocol  (see  Appendix  C)  consisted  of  a 
semi- structured  interview  script  for  the  lead  interviewer  to  follow,  to  ensure  that  all  lines  of 
questioning  were  fully  addressed  within  each  interview.  The  protocol  began  with  a  statement  of 
the  interview  objectives,  followed  by  a  description  of  the  interview  purpose.  The  next  step  in  the 
protocol  was  to  elicit  demographic  infonnation,  and  specific  questions  were  prepared  for  both 
active  duty  and  civilian  personnel.  The  remainder  of  the  protocol,  representing  the 
preponderance  of  the  interview  time,  addressed  three  lines  of  questioning.  The  first  line  of 
questioning  was  site  selection,  specifically,  the  factors  or  characteristics  to  be  considered  when 
slotting  a  task  for  training.  This  section  elicited  data  through  both  general  questions  about  types 
of  tasks  that  should  be  initially  trained  in  institutional,  unit,  and  self-development  settings,  and 
probes  for  specific  incidents  in  which  tasks  were  potentially  assigned  in  a  non-optimal  manner. 
The  next  line  of  questioning  addressed  the  characteristics  of  enduring  and  non-enduring  tasks. 

Its  queries  focused  on  defining  “enduring”  and  “non-enduring”  from  the  interviewee’s 
perspective,  and  determining  the  extent  to  which  the  endurability  of  a  task  is  considered  when 
placing  it  for  training.  In  addition,  the  partial  task  lists  from  a  MOS  relevant  to  the  interviewee 
was  presented  at  this  stage  in  the  interview,  and  participants  were  asked  to  discuss  where  they 
would  recommend  certain  tasks  be  trained,  and  the  rationale  for  those  judgments.  Finally,  the 
last  line  of  questioning  in  the  protocol  addressed  topics  related  to  how  to  best  support  current 
CTSSB  processes  with  an  improved  site  selection  method  and  tool.  These  questions  revolved 
around  who  should  be  the  target  audience  for  the  tool  and  how  the  tool  could  be  used  to  support 
their  site  selection  recommendations. 

Procedure.  The  procedure  included  both  SME  interviews  and  CTSSB  observations. 

Interviews.  Interviews  were  conducted  by  a  research  psychologist  and  supported  by  a 
military  SME  member  of  the  research  team.  In  most  cases,  interviews  were  conducted  with  a 
single  participant;  however,  in  three  cases,  two  SMEs  were  interviewed  together.  Every 
interview  lasted  between  60  and  90  minutes. 

Interviews  began  with  presentation  of  the  Informed  Consent  Form  and  the  Privacy  Act 
Statement.  Once  the  interviewee  read  the  forms  and  signed  his  or  her  consent,  the  interviewer 
requested  verbal  permission  to  digitally  record  the  interview.  Pennissions  were  given  without 
exception,  and  recordings  commenced.  The  data  collection  protocol  was  then  administered, 
beginning  with  the  interviewers  describing  the  purpose  of  the  study  and  clarifying  the 
participant’s  questions  if  necessary,  and  then  collecting  demographic  data  about  the  participant. 
The  majority  of  the  interview  time  was  typically  spent  discussing  the  rationale  for  training  tasks 
at  particular  sites,  the  relative  advantages  and  disadvantages  offered  by  each  site,  and  the 
challenges  associated  with  selecting  sites  for  training,  especially  the  resource  limitations.  In 
addition,  researchers  elicited  information  about  the  CTSSB  process  and  outcomes,  current 
initiatives  for  improvement,  and  other  barriers  associated  with  maintaining  task  lists  and 
updating  institutional  training.  At  the  end  of  each  interview,  participants  were  given  the 
opportunity  to  ask  questions  of  the  research  team,  and  were  provided  contact  information  to  pass 
on  additional  comments  if  desired. 

Observations.  Members  of  the  research  team,  including  two  psychologists  and  one 
military  SME,  conducted  observations  of  two  CTSSBs.  While  the  manner  in  which  these  boards 
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are  conducted  varies  widely,  the  typical  one-week  duration  of  a  CTSSB  was  true  for  boards  we 
observed. 

Researcher  observations  took  place  in  conjunction  with  interviews  at  two  of  the  three 
sites,  and  priority  of  scheduling  always  went  to  the  interviews.  Therefore,  total  observation  time 
amounted  to  approximately  ten  hours  during  the  first  week-long  CTSSB  and  six  hours  during  the 
second  week-long  CTSSB.  In  each  case,  researchers  observed  the  instruction  and  guidance 
provided  to  board  members  by  the  administrators  and  chairmen.  When  possible,  the  researchers 
attempted  to  observe  portions  of  the  CTSSBs  during  which  group  discussion  occurred,  such  as 
during  task  criticality  ratings  or  site  selection.  Notes  were  taken  throughout  the  observation 
process  to  capture  factors  and  task  characteristics  referenced  during  both  task  criticality  and  site 
selection  discussions,  note  questions  and  challenges  identified  by  board  members,  and  document 
the  process  followed  by  the  group. 

Analysis.  Participant  numbers  were  assigned  to  each  interviewee  and  names  were 
removed  from  all  the  data  records.  Notes  were  compiled  for  analysis  and  a  total  of  15  interviews 
were  transcribed.  A  three-stage  qualitative  analysis  process  ensued.  Participant  numbers  were 
associated  with  data  throughout  the  analysis  process  to  ensure  continuous  traceability  back  to  the 
raw  data. 

Analysis  Stage  1  -  Sweep  through  the  Data.  The  primary  research  objective  was  to 
detennine  the  task  characteristics  (including  task  endurability)  and  other  issues  site  selection 
experts  consider  to  distinguish  between  critical  tasks  that  are  ideally  suited  to  training  in  the 
institution,  the  unit,  or  self-development.  In  order  to  answer  the  research  question,  the  traits  that 
make  a  task  best  suited  to  one  training  environment  over  another  must  be  identified.  As  such, 
the  team  judged  that  six  types  of  information  should  be  extract  from  the  interviewee  responses 
and  observations  pertinent  to  site  selection  decisions.  An  additional  four  types  of  infonnation 
were  identified  as  elements  pertaining  to  the  context  in  which  site  selection  recommendations  are 
made,  and  therefore  relevant  to  the  goal  of  developing  a  site  selection  methodology  and  support 
tool.  In  the  first  stage  of  the  analysis,  a  sweep  was  conducted  through  observation  and  interview 
data  to  identify  the  factors  and  task  characteristics  identified  as  pertinent  to  site  selection, 
characteristics  associated  with  enduring  or  non-enduring  tasks,  challenges  associated  with  site 
selection,  ideas  for  improving  site  selection,  and  other  issues  of  relevance  to  the  project  goals. 

For  each  interviewee,  relevant  data  were  extracted  and  entered  into  data  tables  with  the  following 
ten  specific  headers  as  data  groupings: 

1 .  Factors  considered  when  recommending  tasks  for  institutional  training 

2.  Examples  of  tasks  suited  to  institutional  training 

3.  Factors  considered  when  recommending  tasks  for  unit  training 

4.  Examples  of  tasks  suited  to  unit  training 

5.  Factors  considered  when  recommending  tasks  for  self-development 

6.  Examples  of  tasks  suited  to  self-development 

7.  Challenges  associated  with  site  selection 

8.  Challenges  associated  with  detennining  whether  a  task  is  enduring  or  non-enduring 

9.  Other  issues  of  interest  to  the  process 

10.  Ideas  for  supporting  or  improving  the  site  selection  process 
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Analysis  Stage  2  -  Definition  of  Site  Selection  Factors.  Once  the  relevant  data  were 
extracted  from  the  interview  records,  an  informal  thematic  analysis  was  conducted.  Items 
described  as  factors  for  site  selection  (items  1,3,  and  5  in  the  list  above)  were  sorted  into  groups 
of  like  items  as  judged  by  the  research  team.  While  interviewees’  tenninology  varied,  their 
descriptions  of  and  rationale  for  the  factors  and  considerations  reported  conveyed  that 
tenninology  notwithstanding,  similar  concerns,  ideas,  and  considerations  were  applied  to  site 
selection.  The  research  team  ensured  proper  interpretation  of  the  interviewees’  reports  by 
including  the  interviewer,  who  had  first  hand  exposure  to  the  data  collected,  in  the  analysis.  Two 
researchers  conducted  this  thematic  analysis  and  resolved  differences  when  their  groupings  did 
not  agree.  The  data  groupings,  and  the  inclusion  or  exclusion  of  factors  in  each  grouping,  were 
further  verified  by  the  team’s  military  SME.  Each  grouping  was  then  labeled,  resulting  in  a 
single  descriptor  for  the  same  concept  communicated  in  different  ways  by  interviewees.  These 
labels  became  the  36  site  selection  factors.  Frequencies  were  calculated  across  the  data  set  to 
detennine  the  number  of  interviewees  reporting  each  factor,  regardless  of  whether  they  aligned 
the  factor  with  the  institution,  the  unit,  or  self-development.  Therefore,  each  factor  was  stated  as 
a  neutral  element,  such  that  it  could  be  rated  as  having  a  value  across  a  continuum,  or  as  a  yes  or 
no  judgment.  For  instance,  on  the  Universality  factor,  a  task  could  be  judged  as  high,  medium, 
or  low  as  to  the  requirement  for  and  relevancy  to  every  unit  across  the  MOS.  Similarly,  on  the 
Hands-On/Motor  Skills  factor,  a  task  could  be  judged  as  either  requiring  or  not  requiring  hands- 
on  motor  skills  to  perform.  Factors  that  were  not  reported  by  at  least  three  SMEs  were  discarded 
from  the  set  as  non-essential.  Each  remaining  factor  was  then  defined  to  reflect  the  meaning 
conveyed  by  the  associated  data  items. 

Characteristics  of  enduring  and  non-enduring  tasks  were  also  extracted  from  the  data 
tables  and  listed,  using  the  research  team’s  judgment  as  to  data  elements  constituting 
characteristics.  However,  due  to  a  lack  of  SME  support  for  the  application  of  task  endurability 
as  a  site  selection  factor,  few  characteristics  were  proffered  and  so  frequency  counts  for  these 
characteristics  were  not  recorded.  Rather,  the  concept  of  task  endurability  was  more 
appropriately  represented  by  a  subset  of  the  site  selection  factors  already  indicated  by  SMEs, 
most  notably  those  related  to  the  universality,  permanence,  and  foundational  nature  of  tasks. 
Therefore,  the  research  objective  to  specify  characteristics  of  enduring  and  non-enduring  tasks 
was  reframed  as  a  goal  to  identify  the  breadth  of  site  selection  factors  comprising  the  concept  of 
endurability.  Ultimately,  as  described  in  Analysis  Stage  3  below  and  illustrated  in  Table  2, 
“Enduring  Anny  Standards”  was  identified  as  a  category  of  site  selection  factors  representative 
of  the  concept  and  characteristics  of  task  endurability. 

Analysis  Stage  3  -  Factor  Categorization.  In  the  final  analysis  stage,  site  selection 
factors  defined  in  Stage  2  were  further  categorized  by  type/category  of  factor,  for  two  reasons. 
First,  the  large  number  and  diverse  nature  of  the  site  selection  factors  resulting  from  the  analysis 
made  the  set  unwieldy  without  an  organizing  framework.  Some  factors  pertained  to  the 
instructional  setting  that  best  promoted  learning.  Others  related  to  practicalities  such  as  time  and 
resource  availability.  Still  others  had  to  do  with  the  nature  of  the  task  as  a  component  of  the 
MOS.  Second,  organizing  the  factors  into  categories  would  allow  for  highlighting  the 
relationships  among  factors  and  the  clusters  that  often  correlate  with  each  other.  For  example,  a 
task  judged  high  on  the  Universality  factor  was  often  also  judged  high  on  the  Core  Task, 
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Foundational,  and  Need  for  Standardization  factors.  These  factors  all  contributed  to  the 
description  of  a  task  that  is  central  to  the  MOS  and  maintains  criticality  over  the  course  of  time. 
Likewise,  a  task  for  which  Safety  is  a  highly  important  factor  is  often  also  high  on  the  Risk 
factor.  Within  a  category  of  factors,  some  factors  discriminated  between  training  site  in  an 
identical  or  nearly  identical  manner,  and  others  factors  within  a  category  had  unique 
discrimination  patterns  that  provided  unique  insight  to  training  site  selection  above  and  beyond 
the  other  factors  in  the  same  category.  Utilizing  the  category  approach,  factors  that  did  not 
contribute  to  site  determination  via  a  unique  discrimination  pattern  could  be  excluded  from 
consideration  in  the  streamlined  front-end  analysis  methodology. 

The  first  step  in  this  categorization  stage  was  to  sort  the  factors  into  like  groupings.  Two 
researchers  conducted  individual  sorts.  The  factor  definitions  were  closely  examined  throughout 
the  sorting  process.  The  two  researchers  then  compared  their  sorts  and  resolved  their 
disagreements.  Each  grouping  was  given  an  initial  label.  The  categories  and  their  contents  were 
then  presented  to  the  team’s  military  subject  matter  expert  for  his  review.  Once  again, 
disagreements  were  resolved.  A  final  total  of  five  categories  were  defined.  Category  labels  were 
then  finalized  to  maximize  communicability  to  a  military  user  audience. 


Findings 


Site  Selection  Factors  Outcome 

A  total  of  36  site  selection  factors  were  identified  across  five  categories.  They  appear  in 
Table  2,  below,  with  their  definitions.  Table  3  provides  frequency  counts  associated  with  each 
factor,  for  both  Army  and  Coast  Guard  interview  participants. 
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Table  2 

Site  Selection  Factors 


Site  Selection  Factors 

Category 

Factor 

Definition 

Safety 

The  danger  associated  with  training  or  performing  the  task,  and  as  a  result  the  extent  to  which 
safety  measures  must  be  put  in  place  to  avoid  injury. 

Risk 

Whether  significant  risk  (life,  limb,  eyesight)  is  associated  with  incorrect  performance,  either 
in  training  or  operational  contexts. 

Task 

Performance 

Doctrinal  Basis 

The  extent  of  the  requirement  to  train  the  textbook  solution  and  repercussions  of  incorrect 
performance. 

Frequency  of  Performance 

The  regularity  with  which  the  task  will  be  performed. 

Current  Force  Proficiency 

The  degree  of  skill  currently  held  by  NCOs  across  the  Force  or  MOS. 

Unit-Specificity  and 
MOS-Specificity 

The  degree  to  which  the  task  is  driven  by  mission  requirements  of  a  particular  unit,  and 
therefore  subject  to  unit  SOP  vice  generalized  doctrine.  The  degree  to  which  the  task  is 
specific  to  a  particular  MOS  and  not  conducted  by  Soldiers  from  other  MOSs. 

Universality 

The  extent  to  which  the  task  is  required  by  and  relevant  to  units  across  the  Force  or  across  the 
MOS. 

Core  Task 

The  extent  to  which  the  task  is  an  enduring  capability  or  building  block  of  the  MOS,  meaning 
that  all  Soldiers  in  the  MOS  are  expected  to  be  able  to  perform  the  task. 

Need  for  Standardization 

The  importance  associated  with  ensuring  the  task  has  been  instructed  to  a  common  baseline, 
via  a  thorough,  consistent,  and  standardized  manner,  to  each  and  every  NCO.  This  includes 
instruction  regarding  the  importance  of  correct  performance  as  opposed  to  taking  shortcuts  in 
training  the  process. 

Enduring  Army 
Standards 

Need  for  Control 

The  importance  associated  with  ensuring  task  training  has  been  delivered  and  performance 
has  been  assessed  for  each  and  every  NCO. 

Foundational 

The  degree  to  which  the  task  represents  the  basic  building  blocks  of  a  skill  set  or  the 
underlying  knowledge  required  as  context  for  a  larger  set  of  tasks. 

Leadership 

The  importance  or  centrality  of  the  task  for  an  effective  leader. 

Doctrinal 

The  degree  to  which  the  task  relates  to  a  broad  concept  of  battle  that  applies  across  mission 
contexts  vice  a  specific  implementation  of  the  concept  to  a  particular  mission  or  environment. 

Task  Criticality 

The  overall  importance  attributed  to  the  task. 

Train-the-Trainer 

The  extent  to  which  learners  are  expected  to  train  the  task  to  other  Soldiers  at  the  unit. 

Instructional 

Affordances 

Peer  Learning  Benefit 

The  degree  to  which  there  is  a  learning  advantage  associated  with  exposure  to  and  discussion 
of  the  experiences  of  Soldiers  from  other  units  or  backgrounds. 
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Site  Selection  Factors 


Category 

Factor 

Definition 

Small  Group  Instruction 

The  degree  to  which  there  is  a  learning  advantage  associated  with  a  small  student  group 
facilitated  by  an  instructor. 

Question  &  Answer 

The  degree  to  which  students  are  likely  to  require  clarification  of  concepts  during  the 
instruction. 

Practical  Application 

The  degree  to  which  there  is  a  learning  advantage  associated  with  practical  application 
exercises  or  other  hands-on  elements  of  the  instruction. 

Confidence  Building 

The  extent  to  which  learning  outcomes  depend  on  an  instructor  building  learner  confidence. 

Residence  of  Expertise 

The  location  at  which  task  subject-matter  experts  are  most  likely  to  reside,  either  as  a  result 
of  prior  experience  or  doctrinal  study. 

Need  for  Observation-Based 
Assessment 

Whether  performance  must  be  evaluated  against  a  standard  via  observed  performance  versus 
within  a  computer-graded  test,  such  as  a  multiple  choice  exam. 

Site  Affordances 

Need  for  Equipment 

The  degree  to  which  equipment  (weapons,  vehicles,  systems)  is  required  to  train  the  task. 

Access  to  Equipment 

Whether  the  necessary  equipment  for  training  the  task  is  available  at  the  institution  or 
available  at  most  units. 

Time  Available  to  Train 

The  amount  of  time  available  to  train  the  task,  most  often  given  a  limited  course  length. 

Time  Required  to  Train 

The  amount  of  time  necessary  to  provide  sufficient  task  instruction. 

Annual  Training  or 
Qualification 

Whether  or  not  there  is  an  annual  training  or  qualification  requirement  that  is  satisfied  at  the 
unit. 

Integration  Ability 

Whether  the  training  for  the  task  can  be  easily  incorporated  into  training  that  already  exists 
for  other  tasks. 

Training  Updates 

The  likelihood  that  task  standards  will  change  and  instruction  will  require  modification  to 
keep  pace  with  the  evolving  standards. 

Nature  of 
Knowledge 

Hands-On/Motor  Skills 

The  extent  to  which  task  performance  and  training  requires  motor  or  hands-on  performance, 
or  physical  manipulation  of  tools,  equipment,  or  items. 

Basic  Facts  and  Procedures 

Whether  the  task  instruction  is  based  on  declarative  and/or  factual  knowledge  (including 
historical  information  or  procedural  steps),  that  are  relatively  easy  to  learn. 

Task  Complexity 

The  effort  required  to  learn  the  task,  usually  based  on  the  degree  of  difficulty  of  the 
conceptual  or  procedural  knowledge. 

Creative  and 

Whether  the  task  requires  complex  cognitive  skills,  creativity,  or  critical  thinking  to  perform. 
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Site  Selection  Factors 


Category 

Factor 

Definition 

Critical  Thinking 

Detail 

The  degree  to  which  the  task  consists  of  highly  detailed  elements  of  a  process,  procedure,  or 
knowledge  set. 

Segmentability 

The  extent  to  which  the  knowledge  that  comprises  the  task  instruction  can  be  divided  into 
segments  and  instructed  at  different  times. 

Difficulty  to  Retain 

The  degree  to  which  refresher  training  is  required  to  preserve  learner  knowledge  or  skills. 
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Table  3 


Army  and  Coast  Guard  Factor  Frequency  Counts 


Factor 

Army  Frequency 
(n=29) 

Coast  Guard  Frequency 
(n=3) 

Tas 

c  Performance 

Safety 

6 

2 

Risk 

6 

2 

Doctrinal  Basis 

13 

3 

Frequency  of  Performance 

4 

3 

Current  Force  Proficiency 

3 

2 

Unit-Specificity  and  MOS-Specificity 

14 

0 

Enduring  Army  Standards 

Universality 

11 

1 

Core  Task 

12 

0 

Need  for  Standardization 

9 

3 

Need  for  Control 

7 

3 

Foundational 

12 

2 

Leadership 

7 

0 

Doctrinal 

17 

0 

Task  Criticality 

8 

2 

Train-the-Trainer 

2 

0 

Instructional  Affordances 

Peer  Learning  Benefit 

7 

1 

Small  Group  Instruction 

3 

0 

Question  &  Answer 

5 

1 

Practical  Application 

17 

0 

Confidence  Building 

3 

1 

Residence  of  Expertise 

7 

1 

Need  for  Observation-Based  Assessment 

6 

0 

Site  Affordances 

Need  for  Equipment 

12 

2 

Access  to  Equipment 

12 

1 

Time  Available  to  Train 

8 

2 

Time  Required  to  Train 

5 

1 

Annual  Training  or  Qualification 

3 

0 

Integration  Ability 

4 

0 

Training  Updates 

5 

2 

Nature  of  the  Knowledge 

Flands-On/Motor  Skills 

17 

2 

Basic  Facts  and  Procedures 

15 

2 

Task  Complexity 

7 

2 

Creative  and  Critical  Thinking 

11 

1 

Detail 

3 

0 

Segmentability 

5 

0 

Difficulty  to  Retain 

8 

1 
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Task  Endurability 


Task  endurability  was  largely  considered  by  the  SMEs  to  be  the  longevity  of  the  task,  or 
its  likelihood  to  remain  constant  across  time.  Endurability  was  frequently  associated  with  the 
foundational  nature  of  the  task  and  its  universal  application  across  operational  contexts  -  both  of 
which  were  reported  as  site  selection  factors.  Tasks  perceived  to  be  non-enduring  fell  into  one  of 
two  categories.  First,  tasks  can  be  non-enduring  if  they  stem  from  a  particular  adversary  or 
region  of  the  world,  and  TTPs  that  are  produced  as  a  result.  Tasks  can  also  be  non-enduring  if 
they  are  associated  with  a  specific  technology-based  system  (as  opposed  to  the  outcome 
produced  by  the  system)  that  will  be  modified  or  replaced  as  a  result  of  technological  advances. 

The  concept  of  task  endurability  was  considered  by  some  SMEs  to  be  theoretically  sound. 
It  made  logical  sense  that  institutional  resources  should  not  be  applied  to  tasks  whose  criticality 
and  relevance  will  not  persist.  However,  in  practice,  no  SME  reported  having  considered  task 
endurability  in  the  past  as  a  distinct  factor  for  site  selection,  nor  did  SMEs  identify  endurability 
as  particularly  important  when  assessing  the  task  list  provided  as  a  part  of  the  interview  protocol. 

Judgments  about  whether  tasks  are  enduring  or  non-enduring  were  reported  as  complex 
and  difficult,  mainly  due  to  the  fact  that  identifying  enduring  tasks  requires  some  speculation  as 
to  future  world  events.  Several  other  challenges  add  to  the  complication. 

Interview  participants  submit  that  oftentimes  tenninology  changes  to  reflect  a  novel 
instantiation  of  an  old  concept.  For  example,  in  the  last  decade  of  combat,  improvised  explosive 
devices  (IEDs)  have  been  recognized  as  one  of  the  primary  weapons  used  by  the  adversary.  If  we 
take  the  definition  for  task  endurability  related  to  adversary  and  theater-specific  TTP,  then  tasks 
related  to  countering  IEDs  would  be  defined  as  non-enduring.  However,  SMEs  have  argued  that 
IEDs  are  in  actuality  a  new  name  for  what  was  previously  termed  landmines,  as  used  in  the 
Vietnam  era,  and  booby  traps,  as  used  by  countless  other  adversaries.  And,  the  belief  is  that 
future  adversaries  will  also  employ  similar  weapons  and  tactics. 

One  set  of  SMEs  representing  task  analysis  activities  reported  ongoing  efforts  to  modify 
the  task  descriptions  in  a  manner  that  would  minimize  any  discussion  about  pennanent  versus 
temporary,  or  enduring  versus  non-enduring  tasks.  Some  tasks  designate,  in  their  title,  a 
particular  system  or  piece  of  equipment  employed  by  the  NCO.  This  task  analysis  group  argues 
that  what  is  critical  to  perfonnance  is  not  the  particular  tool  applied  so  much  as  the  function  and 
outcome  the  tool  supports.  While  the  impetus  for  their  task  modification  efforts  stems,  in  this 
case,  from  a  lack  of  system  standardization  across  units  staffed  by  NCOs  from  the  MOSs,  the 
point  remains  that  functions  are  more  likely  to  be  enduring  rather  than  tools. 

A  group  of  interviewees  re-framed  the  question  of  enduring  tasks  into  one  of  emerging 
tasks.  An  emerging  task  might  be  defined  as  one  that  materializes  as  a  result  of  global  changes 
to  how  nations  or  other  entities  engage  in  conflict,  or  evolutions  of  political  interests  and  national 
security  policies.  In  contrast  with  the  concept  of  task  endurability,  task  emergence  might  be 
considered  to  be  more  pennanent  in  nature  and  associated  with  a  phase  of  international 
conditions  and  common  practices  in  global  conflict,  vice  a  specific  regional  task  requirement. 
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Some  of  the  participants  in  our  interviews  described  how  the  role  of  our  military  and  elements 
within  the  force  has  evolved  over  time,  thus  producing  a  new  set  of  requirements  that  tend  to 
emerge  over  a  period  not  necessarily  linked  to  a  specific  regional  conflict.  In  the  parlance  of 
military  historians  and  scholars,  the  current  phase  of  conflict  can  be  described  as  “fourth 
generation  warfare”  (Hammes,  2006),  which  includes  elements  of  Vietnam-era  guerilla  fighting 
through  present  day  counterinsurgency  operations  in  Iraq  and  Afghanistan.  Perhaps  the  best 
exemplar  of  this  concept  is  the  changing  role  of  the  infantry  unit  over  the  course  of  the  past 
decade,  from  a  maneuver  warfare  mindset  to  asymmetric  warfare  in  a  counterinsurgency 
environment.  Small  unit  leaders  perform  tactical  questioning  and  other  similar  tasks  related  to 
human  intelligence  gathering,  engagements  with  fonnal  and  informal  civilian  leaders,  and 
biometrics  collections.  These  task  areas  were  previously  primarily  associated  with  intelligence, 
civil  affairs,  and  military  police  MOSs,  respectively,  but  have  emerged  as  infantry  requirements. 
Furthennore,  new  tasks  have  emerged  for  the  Army  as  a  whole  in  the  area  of  site  exploitation 
and  forensics  collection,  prompted  by  both  operational  needs  and  technological  advances.  These 
are  likely  to  reflect  a  semi-permanent  way  of  operating  in  asymmetric  environments.  If  and 
when  these  emerging  tasks  are  classified  as  critical  for  a  particular  MOS  through  the  CTSSB 
process,  the  optimal  training  site  must  be  determined  via  consideration  of  site  selection  factors 
such  as  those  identified  herein. 

Large  degrees  of  variability  existed  in  the  other  comments  made  by  SMEs  regarding 
endurability  as  a  discrete  factor  for  placing  tasks  for  training.  However,  nine  factors  were 
identified  as  related  to  task  endurability,  and  categorized  as  Enduring  Anny  Standards  (see  Table 
2,  above). 


Front  End  Analysis  Methodology  and  Site  Selection  Tool 

Once  the  factors  for  site  selection  were  identified,  the  research  team  produced  a  front-end 
analysis  methodology  for  site  selection  recommendations.  The  methodology  was  then 
instantiated  in  a  SST  that  would  be  highly  accessible  and  immediately  usable  for  CTSSBs.  The 
following  sections  describe  the  development  of  the  FEA,  and  then  describe  the  SST  in  its  current 
iteration.  Interim  versions,  features,  and  assessments  are  described  in  later  sections. 

Front-End  Analysis  Methodology 

Development  of  the  Front-End  Analysis  Methodology.  The  purpose  of  the  FEA 

methodology  was  to  produce  an  efficient,  effective  process,  grounded  in  SME  considerations,  for 
producing  a  training  site  recommendation  based  upon  key  characteristics  of  the  task.  Since  it 
would  have  been  futile  to  produce  a  methodology  that  produced  the  “right”  site  selection  in 
every  instance,  the  design  of  the  methodology  was  instead  driven  by  the  need  for  a  streamlined 
process  that  would  produce  a  valid  recommendation  for  most  all  tasks  except  under  outlier 
circumstances.  With  an  aim  to  define  a  highly  efficient  methodology,  therefore,  only  the  factors 
that  are  principal  discriminators  for  site  placement  were  included  in  the  methodology.  Each  of 
the  five  factor  categories  was  reviewed  to  identify  the  factors  that  appear  to  be  highly  related 
(e.g.,  a  task  that  is  high  on  Universality  is  often  also  high  on  Need  for  Standardization )  and  thus, 
for  the  purposes  of  the  methodology,  redundant.  In  addition,  factors  that  were  seen  as  critical  and 
key  determinants,  such  as  Safety  or  Peer  Learning  Benefit,  were  identified  as  candidate 
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discriminators  for  the  methodology.  To  identify  these  candidates,  the  research  team  and  the 
team’s  military  SME  made  the  initial  judgment  as  to  which  factors  provided  strong  and  clear 
discrimination  between  tasks  suited  for  institution,  unit,  or  self-development  training  sites. 
Factors  that  made  finer  discriminations  that  were  less  clear,  made  discriminations  between  a 
minority  of  tasks,  or  were  redundant  with  each  other  were  deemed  of  lower  value  to  the 
methodology.  For  example,  safety  is  always  the  utmost  concern  for  the  Anny  whenever  training 
or  missions  are  concerned.  Safety  concerns  can  also  easily  distinguish  between  tasks  suited  for 
independent  self-study  without  SME  cautionary  oversight,  and  tasks  requiring  the  close  watch  of 
an  expert  trainer.  As  such,  Safety  was  considered  a  highly  important  factor  in  the  FEA 
methodology.  The  Risk  factor  is  similar  enough  to  Safety  to  be  considered  redundant,  and 
therefore  was  excluded  from  the  methodology  as  a  independent  discriminator.  (However,  Safety 
and  Risk  were  later  merged  into  a  single  discriminator  in  the  FEA  methodology.) 

Finally,  factors  deemed  difficult  for  a  CTSSB  board  member  to  assess  due  to  their  limited 
knowledge  of  the  institution’s  POI  and  resources,  such  as  Time  Available  to  Train  or  Integration 
Ability,  were  excluded  as  candidates.  An  initial  list  of  candidate  principal  discriminators  was 
prepared.  This  process  was  aided  by  the  inputs  of  the  military  SME  who  was  a  member  of  the 
research  team. 

A  flow  chart  diagram  was  iteratively  generated  along  with  the  identification  of  candidate 
discriminators.  The  flow  chart’s  logic  chain  was  determined  by  identifying  the  factors  that  were 
highly  discriminatory  (i.e.,  making  large  distinctions  between  training  sites,  like  Safety)  and 
those  that  were  moderate  in  discriminatory  ability  (i.e.,  making  moderate  distinctions  and 
justifications  between  training  sites).  Those  with  higher  discrimination  capabilities  were 
included  early  in  the  logic  flow;  factors  deemed  moderate  in  their  discriminatory  ability 
followed.  In  addition,  some  factors  were  included  later  in  the  logic  chain  (or  possibly  skipped 
altogether  depending  upon  the  particular  path  a  user  follows),  if  the  factor  was  seen  as  irrelevant 
based  upon  answers  to  earlier  questions.  For  example,  when  considering  whether  a  task  should 
be  trained  via  self-development,  if  a  task  requires  access  to  actual  (and  not  simulated)  equipment 
access,  then  the  task  is  a  poor  fit  for  self-development/dL  (as  equipment  would  be  difficult  to 
attain  for  self-development  and  dL).  In  this  example,  asking  whether  a  task  is  too  difficult  to  be 
learned  via  self-development  is  a  moot  point  because  the  complexity  of  the  task  is  a  secondary 
consideration  to  whether  or  not  the  training  can  be  resourced  via  self-development.  This  same 
logic  approach  of  prioritizing  factors  was  considered  for  the  development  of  the  entire  flow 
chart.  The  flow  chart  logic  chain  was  vetted  and  revised  (as  described  later  in  this  report) 
through  the  team’s  military  SME,  the  research  sponsor,  training  development  and  CTSSB  SMEs, 
and  the  assessment  and  validation  phases  of  this  research  effort. 

In  some  cases,  two  discriminating  factors  were  combined  to  enhance  the  clarity  or  add 
emphasis  to  the  discrimination  the  eventual  user  would  be  asked  to  make.  Each  candidate 
discriminator  or  pair  of  discriminators  was  re-framed  as  a  yes/no  question  to  be  answered  by  the 
user  about  the  task,  and  represented  in  the  flow  chart  as  such.  Affirmative  or  negative  responses 
would  define  the  branch  to  the  next  discriminator  in  the  chart.  Later  in  the  development  process, 
following  user  feedback,  the  affirmative  and  negative  response  options  were  modified  to  allow 
for  responses  along  a  continuum  -  high,  medium,  or  low  -  for  a  subset  of  the  discriminators. 

The  flow  chart  was  conceptualized  to  begin  with  a  series  of  questions  designed  to  first 
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recommend  or  eliminate  self-development  as  the  training  site;  then  recommend  or  eliminate  the 
institution  as  the  site;  and  finally  recommend  the  unit  if  the  other  two  sites  were  eliminated,  and 
if  the  unit  was  equipped  to  provide  the  training. 

Early  versions  of  the  flow  chart  diagram  were  vetted  by  the  research  team  and 
stakeholders  to  assess  the  face  validity  of  the  discriminators  included  and  excluded  (i.e.,  were 
obvious  important  factors  excluded  or  missing),  and  to  gauge  the  suitability  of  the  rules  for 
branching  from  one  question  to  the  next  and  finally  to  a  site  recommendation.  In  addition,  a 
series  of  NCO  tasks  taken  from  multiple  MOSs  were  subjected  to  the  flow  chart  algorithm  by  the 
team’s  military  SME  to  examine  the  legitimacy  of  the  site  recommendations  produced  when 
applying  the  methodology.  The  discriminating  factors,  wording  of  questions,  and  branching 
rules  were  modified  as  a  result  of  these  internal  reviews. 

Front-End  Analysis  Methodology  Outcome.  Fourteen  factors  were  ultimately 
identified  as  principal  discriminators  for  site  selection  and  articulated  in  a  series  of  twelve 
questions  in  the  methodology.  Table  4  identifies  the  discriminators,  how  they  were  combined 
and  reframed  within  the  FEA  methodology,  the  short  and  long  versions  of  the  question  (for  space 
considerations  in  the  eventual  SST),  and  their  response  options. 


24 


Table  4 


Discriminators  of  the  Front-End  Analysis  Methodology 


Factor 

Short  Question 

Long  Question 

Response 

Options 

Definition 

Safety 

Safety  Concern 

DURING  TRAINING,  what 
is  the  risk  of  injury  to 
personnel  or  damage  to 
equipment  when  training  this 
task? 

High,  Medium, 
Low 

The  danger  associated  with  training  or 
performing  the  task.  Significant  risk  (life,  limb, 
eyesight)  may  be  linked  with  incorrect 
performance,  either  in  training  or  operational 
contexts.  This  question  is  about  the  degree  of 
risk  associated  with  conducting  the  task. 

Risk 

Need  for  Equipment 

Equipment  and 
hands-on  training 
required? 

Is  actual  equipment  -  a 
weapon,  vehicle,  or  system  - 
and  hands-on  training, 
instead  of  an  equipment 
substitute,  required  to  train 
this  task? 

Yes,  No 

The  extent  to  which  task  performance  and 
training  requires  motor  or  hands-on 
performance,  or  physical  manipulation  of  tools, 
equipment,  or  items.  This  question  is  about 
whether  the  training  must  employ  the  actual 
equipment  that  would  be  used  operationally,  or 
whether  computer  based  training,  a  simulation, 
or  a  classroom-based  substitute  can  be  used 
instead. 

Hands-on  Motor  Skills 

Task  Complexity 

Difficult  to  learn? 

How  complex  or  difficult  to 
learn  is  this  task? 

High,  Medium, 
Low 

The  effort  required  to  learn  the  task,  usually 
based  on  the  degree  of  difficulty  of  the  concepts 
or  procedures  involved.  Tasks  that  are  more 
complex  require  a  human  trainer  for  instruction, 
to  provide  examples,  and  to  answer  questions. 
This  question  is  about  the  degree  of  task 
complexity. 

Need  for  Observation- 
Based  Assessment 

Assess 

performance  by 
watching  face-to- 
face? 

Does  a  trainer  need  to  watch 
performance  face-to-face  to 
assess  it? 

Yes,  No 

The  extent  to  which  a  Soldier’s  performance 
must  be  observed  in  order  to  evaluate  it  against 
the  standard.  This  question  refers  to  whether  a 
trainer  is  required  to  watch  NCOs  perform  the 
task,  in  person,  in  order  to  conduct  an  accurate 
assessment. 

Training  Updates 

Updates  needed 
more  than 
yearly? 

Will  the  training  change 
more  than  once  a  year? 

Yes,  No 

The  likelihood  that  task  standards  will  change 
and  instruction  will  require  modification  to 
keep  the  pace  with  the  evolving  conditions.  If 
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Factor 


Short  Question 


Long  Question 


Response 

Options 


Definition 


the  training  content  will  change  more  than  once 
a  year,  the  task  is  probably  not  a  good  fit  for 
self-development. 

Residence  of  Expertise 

SMEs  available 
or  accessible  at 
unit? 

Are  SMEs  readily  available 
or  accessible  to  train  this 
task  at  the  unit? 

Yes,  No 

The  extent  to  which  trainers,  small  unit  leaders, 
or  subject  matter  experts  are  training  the  task  at 
the  unit.  This  question  refers  to  whether  subject 
matter  expertise  typically  exists  at  the  unit, 
whether  those  SMEs  are  readily  available  to 
NCOs  for  training,  or  whether  SMEs  can  be 
easily  brought  to  the  unit. 

Annual  training  or 
qualification 

Annual 
qual./cert.  at 
unit? 

Is  recurring  qualification  or 
certification  conducted  at 
unit? 

Yes,  No 

Whether  or  not  there  is  an  annual  training  or 
qualification  requirement  that  is  satisfied  at  the 
unit.  The  rationale  behind  this  question  is  that 
when  annual  testing  occurs  at  the  unit,  there  is  a 
strong  likelihood  that  NCOs  receive  the 
required  training  at  the  unit. 

Universality 

Universal  task, 
uniform  training 
required? 

Is  the  task  universal  and 
requiring  uniformity  of 
training  across  the  MOS? 

Yes,  No 

This  question  has  two  parts.  Universality  is 
about  the  extent  to  which  the  task  is  relevant  to 
NCOs  across  the  Force  or  across  the  MOS, 
regardless  of  job  assignment.  Standardization 
refers  to  how  important  it  is  that  all  Soldiers 
learn  to  conduct  the  task  using  the  same 
procedure.  This  question  asks  whether  most 
NCOs  will  employ  the  task  on  the  job,  and 
whether  they  must  know  or  use  the  book 
standard  to  be  effective. 

Need  for 

Standardization 

Need  for  Control 

Reliable  and 

controlled 

assessment? 

Is  a  reliable  and  controlled 
assessment  highly 
important? 

Yes,  No 

The  importance  associated  with  ensuring  task 
training  has  been  delivered  and  performance  has 
been  assessed  for  each  and  every  NCO.  Some 
tasks  require  an  extra  degree  of  assurance  that 
every  Soldier  who  has  gone  through  the  training 
actually  grasps  the  task  and  can  perform  it 
effectively.  This  question  refers  to  the 
criticality  that  an  instructor  be  present  to  ensure 
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Factor 

Short  Question 

Long  Question 

Response 

Options 

Definition 

each  and  every  NCO  is  trained  to  standard. 

The  degree  to  which  there  is  a  learning 
advantage  associated  with  exposure  to  the 
experiences  of  Soldiers  from  other  units  or 
backgrounds.  When  a  task,  such  as  a  combat 

Peer  Learning  Benefit 

Peer-to-peer 
learning  critical 
and  required? 

Is  sharing  experiences  with 
peers  from  other  units 
critical  and  required  to 
maximize  learning? 

Yes,  No 

tactic,  must  be  adjusted  for  execution  in 
different  mission  types  or  operational 
environments,  there  is  great  learning  value  in 
hearing  examples  of  the  many  ways  to  conduct 
a  task.  This  question  refers  to  whether  there  is  a 
particular  benefit  from  exposure  to  peers  from 
different  units  who  can  speak  to  how  mission  or 
environmental  factors  impact  task  execution. 

Leadership 

Criticality  to 
leadership? 

How  critical  is  the  task  to  the 
leadership  role? 

High,  Medium, 
Low 

The  importance  or  centrality  of  the  task  for 
being  an  effective  leader.  This  question  is 
about  whether  or  not  the  task  is  one  of  the  core 
functions  of  a  leader. 

Access  to  Equipment 

Safety/equipment 
available  at  unit? 

Do  most  units  have  the 
safety  measures  and 
equipment  readily  available 
to  perform  training? 

Yes,  No 

Whether  the  necessary  equipment  or  safety 
measures  for  training  the  task  are  present  and 
available  at  most  units.  This  question  provides 
a  check  to  ensure  that  if  a  task  is  recommended 
for  training  exclusively  at  the  unit,  most  units 
across  the  Amy  will  have  the  means  to  conduct 
that  training. 
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The  flow  chart  diagram  depicting  the  final  FEA  methodology,  using  the  questions  in 
Table  4  above,  is  shown  in  Figure  3.  The  methodology  is  intended  to  be  applied  to  each  task 
individually.  Users  begin  with  Question  1 ,  regarding  the  Safety  discriminating  factor,  and 
progress  through  the  flow  chart  based  on  their  responses  to  each  question  as  defined  by  the 
arrows. 


Figure  3.  Front-end  analysis  methodology 

Once  satisfied  that  the  FEA  methodology  illustrated  in  the  flow  chart  diagram 
represented  a  sound  process  that  could  be  demonstrated  to  SMEs  for  their  feedback,  the  team 
initiated  development  of  the  SST. 

Site  Selection  Tool 


Development  of  the  Site  Selection  Tool.  The  primary  objective  of  the  SST  was  to 
support  CTSSB  members,  who  are  novice  site  selectors,  in  making  better  use  of  their  operational 
experience  to  make  robust  site  selection  recommendations.  Therefore,  the  FEA  methodology 
was  instantiated  into  the  SST  with  the  goal  of  seamless  integration  of  the  tool  into  the  CTSSB 
context  and  process.  The  SST  was  not  envisioned  as  a  replacement  for  the  judgment  of  the  board 
members.  Instead,  the  goal  was  to  improve  the  quality  of  human  recommendations  by  ensuring 
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all  the  important  factors  are  considered. 

The  current  site  selection  process,  as  observed  by  the  research  team,  entails  a  group 
discussion  of  site  placement  for  those  tasks  identified  by  the  board’s  vote  as  critical.  However, 
the  discussion  is  typically  driven  by  a  combination  of  gut  instinct  and  current  practice  (i.e., 
“where  are  we  currently  training  this  task?”)  to  reach  a  site  recommendation.  Furthennore,  the 
group  dynamic  may  be  such  that  one  or  two  individuals  monopolize  the  discussion  by  virtue  of 
their  assertiveness.  Since  each  individual  represents  one  unit,  the  result  may  be  limited 
contribution  or  no  input  at  all  from  key  operational  elements  whose  agents  are  present  at  the 
board  meeting,  but  who  are  overridden  by  more  assertive  personalities  of  representatives  from 
other  units.  Therefore,  a  secondary  objective  of  the  site  selection  methodology  and  tool  was  to 
enable  equal  voting  on  site  selection  from  all  board  members,  just  as  is  the  case  with  the  task 
criticality  ratings.  The  SST  was  not  envisioned  as  a  means  of  bypassing  the  group  discussion  of 
task  placement.  In  contrast,  we  aimed  to  improve  the  efficiency  of  the  group  discussion  by 
clearly  differentiating  the  tasks  on  which  board  members  agree  and  thus  require  no  discussion, 
from  those  tasks  for  which  there  is  variability  across  board  member  opinions  and  thus  do  require 
discussion. 

The  SST  development  phase  was  initiated  by  identifying  a  medium  that  would  best 
support  both  CTSSB  members  and  the  administrators  of  CTSSBs.  Web-based  tools  and  survey 
platforms  were  considered  for  their  potential  to  be  accessible  from  any  Army  computer  via  Anny 
Knowledge  Online  (AKO).  However,  we  ultimately  selected  Microsoft  Excel  as  the  platform  in 
which  to  build  the  tool.  Excel  is  currently  used  by  most  CTSSBs  to  manage  the  criticality 
ratings.  Therefore,  CTSSB  administrators  are  familiar  with  the  software’s  functionality,  task  list 
inputs  would  only  require  a  simple  copy  and  paste,  and  an  Excel  tool  is  conceptually  consistent 
with  the  manner  by  which  CTSSBs  are  conducted.  Furthennore,  a  product  built  in  Excel  would 
not  be  subject  to  restrictions  on  software  that  can  be  installed  on  Anny  computers,  and  would 
thereby  be  immediately  accessible  to  the  target  audience.7  The  SST  was  programmed  using 
Microsoft’s  programming  tool,  Visual  Basic  2010.  The  resulting  user  experience  was  one  in 
which  the  SST  automatically  and  differentially  produced  outputs  in  response  to  user  inputs  in  a 
manner  similar  to,  but  more  advanced  than,  Excel’s  fonnula  capability. 

Two  versions  of  the  SST  were  created.  A  User  version  of  the  SST,  in  the  fonn  of  a 
macro-enabled  spreadsheet,  was  created  to  lead  board  members  through  the  series  of  questions 
comprising  the  FEA  methodology,  and  resulted  in  an  SST  recommendation  based  on  the  flow 
chart  branching  logic.  Users  were  also  queried  for  their  agreement  with  the  SST 
recommendation  and  their  personal  site  selection  recommendation,  if  it  differed  from  the 
system’s  recommendation. 

An  Administrator  version  of  the  SST  was  also  created.  This  version  enabled  the  CTSSB 
administrator  first  to  input  the  finalized  critical  task  list  into  the  SST  and  disseminate  it  across 
CTSSB  members  (i.e.,  the  computers  they  would  use  in  an  Army  computer  lab).  Then, 


7  Newly  developed  software  is  subject  to  a  review  process  to  obtain  a  Certificate  of  Networthiness  (CON). 
Obtaining  a  CON  is  a  prolonged  process  of  a  year  or  more,  and  updated  CONs  must  be  periodically  obtained  upon 
expiration  of  the  original.  By  building  the  SST  in  a  recognized  Army  software  platform  (i.e.,  Excel),  the  SST  falls 
under  the  existing  CON. 
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subsequent  to  user  completion,  the  administrator  would  merge  the  data  of  the  board  members 
and  create  a  compiled  set  of  recommendations.  As  part  of  the  compilation  function,  the  SST 
would  also  highlight  those  tasks  for  which  board  member  consensus  did  not  achieve  a  desired 
threshold  (i.e.,  50%,  60%,  70%,  or  80%,  as  selected  by  the  Administrator).  Once  user  data  are 
compiled  and  tasks  highlighted  for  discussion,  the  Administrator  version  can  be  viewed  by  either 
the  board  as  a  group,  or  by  the  board  chair  alone,  to  facilitate  discussion  on  low-consensus  tasks. 
Both  User  and  Administrator  versions  of  the  SST  functionality  and  interface  were  modified  as  a 
result  of  four  rounds  of  feedback  (described  in  Site  Selection  Tool  Assessment). 


Site  Selection  Tool  Outcome.  The  User  version  of  the  SST  is  depicted  in  Figure  4. 
Critical  tasks  and  their  numeric  designators  are  inserted  by  the  administrator  in  the  first  two 
columns  of  the  spreadsheet.  The  third  column  (Column  C)  is  vacant  for  the  criticality  rating  to 
likewise  be  entered  by  the  administrator,  if  so  desired;  however,  this  column  is  optional.  By 
default,  this  column  is  hidden  and  is  not  visible  in  Figure  4.  The  next  12  columns  comprise  the 
questions  to  be  answered  by  the  user  about  the  task  in  question.  They  reflect  the  12  principal 
discriminator  questions  from  the  FEA  methodology.  As  illustrated,  the  short  version  of  the 
question  is  viewable  at  all  times.  Users  can  hover  the  mouse  over  the  red  caret  (triangle  in  upper 
comer  of  the  cell)  to  view  the  long  version  of  the  question.  The  detailed  definitions  of  each 
question  are  available  to  users  as  a  separate  tab  in  the  Excel  workbook,  and  included  in 
Appendix  D. 
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Figure  4.  Interface  of  the  board  member  SST 
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The  branching  algorithm  depicted  above  in  Figure  4  is  replicated  in  the  SST  by  graying 
out  the  cells  for  which  an  answer  is  not  required.  For  example,  if  a  user  selects  “high”  or 
“medium”  on  Question  1,  then  Questions  2-5  will  be  grayed  out  and  the  cursor  will  appear  in  the 
box  for  Question  6.  Once  the  user  has  input  an  answer  to  each  question  required  for  a  given  task 
according  to  the  branching  algorithm,  the  SST  will  automatically  populate  the  “SST 
Recommendation”  cell  with  one  of  three  possible  outcomes:  Institution,  Unit,  of  Self¬ 
development.  If  the  user  disagrees  with  the  SST  recommendation,  he  or  she  may  identify  a 
different  site  recommendation  in  column  S  (Board  Member  Opinion),  including  a  selection  of 
“Institutional-Functional”  to  reflect  the  desire  for  the  task  to  be  trained  in  an  institutional  setting, 

o 

but  in  a  functional  course  vice  the  course  being  reviewed.  A  Reset  button  is  provided  in 
Column  R  to  enable  the  user  to  clear  the  row  of  answers  if  he  or  she  wishes  to  make  adjustments 
to  a  response  other  than  the  last  question  answered.  Finally,  users  may  choose  to  insert 
comments  about  the  task  in  the  final  column,  if  desired  (see  Figure  5). 

Task  List  ^ST  User  Opinion,  If  User  Comments 

Recommendation  Different  from  SST 


Figure  5.  User  task  comments  in  SST 

The  Administrator  version  of  the  SST  (see  Figure  6)  compiles  the  data  from  the 
completed  User  forms  using  macros  created  in  Visual  Basic  and  provides  two  frequency  counts 
for  site  placement  votes:  one  for  the  SST  Recommendations  and  one  for  the  Board  Member 
Opinions  (Column  S  from  the  Board  Member  SST  version).  Prior  to  compilation,  the 


8  The  “Institutional-Functional”  site  recommendation  was  added  to  account  for  tasks  that  may  not  be  best  suited  for 
a  formal  NCOES  course  (i.e.,  WLC,  ALC,  SLC),  but  which  board  members  still  believed  should  be  taught  by  the 
institution/schoolhouse.  “Institutional-Functional”  is  not  an  option  that  the  SST  will  recommend,  but  can  be 
selected  by  the  board  member  in  the  personal  opinion  column. 
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administrator  may  designate  a  threshold  for  consensus  to  be  applied  by  the  SST  -  50%,  60%, 
70%,  or  80%.  These  choices  are  provided  to  ensure  the  SST  is  of  value  to  CTSSBs  regardless  of 
the  number  of  voting  members  or  other  group  differences  that  may  call  for  higher  or  lower 
agreement  thresholds.  When  consensus  across  the  board  members  does  not  reach  the  threshold 
in  either  of  the  consensus  columns  (i.e.,  SST  Recommendation  or  Board  Member  Opinion),  the 
row  is  highlighted  amber.  The  highlighting  designates  the  task  to  the  Board  Administrator  as 
requiring  group  discussion,  either  on  the  basis  of  low  consensus  with  the  SST’s  recommendation, 
or  on  the  basis  of  low  consensus  across  the  board  members’  personal  opinions.  In  the  fonner 
case,  the  Board  Member  Opinion  column  may  assist  in  resolving  the  site  selection 
recommendation.  In  the  latter  case,  the  SST  potentially  has  revealed  a  difference  in  training 
needs  across  units  as  represented  by  the  board  members,  which  is  deserving  of  group  dialogue. 
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Figure  6.  Interface  of  the  administrator  SST 


SST  Implementation  Guidance.  To  support  implementation  of  the  SST  during 
CTSSBs,  additional  materials  were  produced.  Step-by-step  instructions  for  the  administrator  set 
up,  board  member  execution,  and  administrator  data  compilation  are  available  (see  Appendix  E). 
To  ensure  the  intended  interpretation  of  each  principal  discriminator  in  the  series  of  questions,  a 
reference  sheet  was  prepared  to  include  detailed  definitions  and  rationale  behind  each  question  in 
the  SST  (see  Appendix  D).  A  short  set  of  recommendations  for  the  CTSSB  administrator  was 
also  created  to  aid  in  the  implementation  of  the  SST  within  the  CTSSB  process.  In  some 
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circumstances,  a  CTSSB  must  be  conducted  in  a  secure  facility  where  the  critical  task  list  may 
not  be  input  into  a  non-secure  computer  terminal.  Under  these  circumstances,  the  paper-based 
flow  chart  may  be  used  as  a  substitute  to  the  SST.  Thus,  instructions  have  also  been  prepared 
with  recommendations  for  employing  the  FEA  methodology  absent  the  SST. 

Site  Selection  Tool  Assessment 

The  final  version  of  the  SST,  as  described  in  the  previous  section,  underwent  a  series  of 
assessment  studies  to  establish  its  usability  and  validity,  by  collecting  user  feedback  and  refining 
the  tool  accordingly.  In  this  section,  we  describe  in  detail  the  assessment  studies  conducted 
concurrently  with  the  SST  development  iteration  efforts. 

The  assessment  of  the  SST  examined  both  the  usability  of  the  system  and  the  face 
validity  of  its  site  selection  recommendations,  by  piloting  the  tool  in  the  context  of  actual 
CTSSBs.  Observations  and  feedback  were  collected  from  the  target  user  groups  -  CTSSB  board 
members  and  administrators.  By  evaluating  the  SST  in  its  intended  use  setting,  we  were  also 
able  to  examine  its  ability  to  integrate  with  the  broader  CTSSB  process,  and  modify 
implementation  recommendations  to  users  accordingly.  All  assessment  activities  were 
conducted  iteratively  with  the  SST  development,  such  that  user  inputs  in  each  round  of 
assessment  infonned  revisions  made  to  the  subsequent  version  of  the  tool.  The  formal 
assessment  sessions  totaled  three. 

Pre- Assessment 

Prior  to  the  formal  evaluation  sessions,  pre-assessment  activities  were  conducted.  The 
primary  purpose  of  these  activities  was  to  obtain  SME  feedback  regarding  the  logic  chain  and 
branching  mechanisms  in  the  FEA  methodology  and  User  SST,  to  ensure  the  face  validity  of  the 
tool’s  site  recommendations.  Four  telephonic  feedback  sessions  were  conducted  with  SMEs 
previously  interviewed  during  the  development  phase.  The  SMEs  received  a  briefing  on  the 
discriminators  and  the  flow  chart  diagram,  then  viewed  a  demonstration  of  the  SST.  Partial  task 
lists  representing  the  SME’s  MOS  were  input  into  the  SST,  and  the  SMEs  responded  to  the 
questions  to  produce  an  SST  recommendation  for  each  task.  The  participants  were  thus  able  to 
judge  the  SST  recommendation  in  comparison  to  their  own  judgment  of  site  placement  for  a 
given  task,  as  a  means  of  assessing  the  quality  of  the  branching  logic.  As  a  result  of  these 
sessions,  the  following  modifications  were  made: 

•  The  terminology  of  some  questions  was  altered  to  ensure  its  intended  meaning  for  the 
military  user  audience. 

•  Since  SMEs  judged  it  too  difficult  to  answer  Questions  1  (Safety  concern?),  3 
(Difficult  to  learn?),  and  1 1  (Critical  to  leadership?)  in  the  affirmative  or  negative,  the 
response  choices  for  these  queries  were  changed  from  “yes”  or  “no”  to  “high,” 
“medium,”  or  “low.”  A  response  of  “high”  or  “medium”  was  then  equated  to  a  “yes” 
response  in  the  branching  logic,  and  a  response  of  “low”  was  equated  to  a  “no” 
response. 

•  The  consensus  threshold  was  set  to  highlight  red  those  tasks  with  0-69%  agreement, 
and  to  highlight  amber  those  tasks  with  70-79%  agreement.  Tasks  with  80%  or 
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higher  agreement  would  not  be  highlighted,  indicating  no  need  to  discuss  the  site 
recommendation. 

•  At  this  stage  of  development,  tasks  would  be  highlighted  when  certain  response 
combinations  were  met  that  suggested  a  questionable  site  recommendation.  For 
instance,  if  users  indicated  that  a  task  was  universal  and  required  unifonnity  of 
training,  and  that  it  benefitted  from  peer  learning,  but  other  responses  slotted  the  task 
for  unit  training,  that  task  row  would  be  highlighted  to  prompt  a  discussion  about 
whether  it  might  be  better  suited  for  the  institution.  As  a  result  of  the  pre-assessment 
feedback  sessions,  two  sets  of  response  combinations  were  dubbed  as  producing 
recommendations  deserving  of  group  discussion.  Upon  further  evaluation  in  Round 
One,  however,  this  functionality  was  abandoned  altogether  in  favor  of  soliciting 
board  member  opinions  in  addition  to  the  SST  recommendation. 

Assessment  Study  One 

The  first  round  of  assessment  was  conducted  in  conjunction  with  the  19A  (i.e.,  Annor 
Officer)  CTSSB  at  Fort  Benning,  GA,  reviewing  both  LT  and  OPT  tasks  for  the  Annored  Officer 
Basic  Leader  Course  (ABOLC)  and  the  Maneuver  Captains’  Career  Course,  respectively. 
Although  the  SST  was  developed  based  upon  NCOES  requirements,  scheduling  constraints 
precluded  waiting  for  an  NCO  board.  The  research  team  and  sponsor  judged  the  CTSSB  process 
to  be  sufficiently  similar  between  NCO  and  Officer  boards  to  allow  for  evaluation  of  the  SST 
integration  with  the  CTSSB  procedure.  Moreover,  at  Fort  Benning,  both  the  Armor  Officer  and 
NCO  boards  are  conducted  by  the  same  organization  utilizing  the  same  procedures,  regardless  of 
whether  the  tasks  are  intended  for  Officers  or  NCOs.  This  practice  is  common  across  the  Anny. 
In  light  of  the  differences  between  entry  level  officer  tasks  and  mid-level  NCO  tasks,  we  were 
prepared  to  see  the  preponderance  of  the  tasks  skewed  toward  institutional  training,  reflecting  a 
need  for  the  Army  educational  system  to  formally  train  basic  skills  before  sending  Officers  to 
their  first  unit. 

The  assessment  study  design  included  three  components:  (1)  comparison  of  pre-SST  task 
placement  to  SST  and  post-SST  placement;  (2)  comparison  of  SST  location  recommendations  to 
human  location  recommendations;  and  (3)  user  feedback.  User  reaction  surveys  were  collected 
from  CTSSB  board  members  and  the  chairman,  and  feedback  was  collected  from  board 
members,  the  board  chair,  and  administrators  in  the  context  of  a  post-test  focus  group. 

Participants.  Eight  members  comprised  the  19A  board.  They  were  a  mix  of  CPTs  and 
Majors  (MAJs)  representing  cavalry,  annor,  and  infantry  units,  the  National  Guard,  and  two  non¬ 
units.  The  SST  was  tested  by  all  eight  board  members  and  the  Lieutenant  Colonel  (LTC)  who 
chaired  the  board. 

Materials.  The  assessment  was  conducted  using  an  Informed  Consent  Form,  Privacy 
Act  Statement,  User  Reaction  Survey,  Focus  Group  Protocol,  and  both  User  and  Administrator 
versions  of  the  SST. 

Informed  Consent  Form.  The  Informed  Consent  Form  (see  Appendix  A)  introduced  the 
purpose  of  the  study,  described  the  assessment  objectives,  clearly  indicated  that  participation  was 
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voluntary  and  could  be  tenninated  at  any  time,  and  assured  participants  that  their  responses 
would  not  be  attributable  to  them  personally.  In  addition,  points  of  contact  were  provided  for 
questions  concerning  the  study  and  questions  about  individuals’  rights  as  interviewees.  The 
participant  signed  the  consent  form  to  acknowledge  that  the  individual  was  volunteering  to 
participate,  was  at  least  1 8  years  of  age,  and  was  willing  to  have  his  or  her  feedback  digitally 
recorded,  if  necessary.  Participants  retained  the  Consent  Fonn  for  their  records,  while  the 
researchers  retained  the  signature  pages. 

Privacy  Act  Statement.  Similar  to  the  Infonned  Consent  Fonn,  the  Privacy  Act 
Statement  (see  Appendix  B)  summarized  the  purpose  of  the  assessment  and  reminded 
participants  of  their  rights  to  confidentiality  and  voluntary  participation.  Individuals  retained  a 
copy  of  the  Privacy  Act  Statement  for  their  records. 

User  Reaction  Survey.  The  User  Reaction  Survey  (see  Appendix  F)  consisted  of  two 
pages.  In  Part  One,  participants  answered  13  questions  using  a  five-point  Likert  scale  where  1  = 
strongly  disagree  and  5  =  strongly  agree.  Queries  addressed  the  SST’s  overall  value,  clarity,  ease 
of  use,  and  impact  on  the  site  selection  process.  In  Part  Two,  four  open-ended  questions  elicited 
user  opinions  regarding  the  SST’s  valuable  features,  distracting  or  irrelevant  features,  change 
requirements,  and  integration  with  the  CTSSB  process. 

Focus  Group  Protocol.  The  Focus  Group  Protocol  (see  Appendix  G),  initially  prepared 
as  a  protocol  for  individual  post-SST  interviews,  was  created  to  elicit  more  detailed  feedback  and 
rationale  for  suggested  SST  modifications  via  discussion  of  the  users’  experience  with  the  tool. 
Lines  of  questioning  focused  on  the  usability  of  the  tool  and  its  functionality,  the  clarity  and 
interpretation  of  the  questions,  and  suggestions  for  including  it  as  part  of  the  CTSSB  process. 

Site  Selection  Tool.  The  User  and  Administrator  SSTs  were  loaded  onto  computers  in  a 
computer  lab  in  close  proximity  to  the  CTSSB  meeting  room.  A  subset  of  the  19A  LT  critical 
tasks  was  input  into  the  tool  prior  to  testing. 

Procedure.  Prior  to  the  board,  an  Annor  SME  from  the  research  team  generated  two 
lists  of  80  19A  tasks  that  were  roughly  matched  in  terms  of  topics  and  content.  The  intention 
was  to  create  two  task  lists  containing  representation  of  each  of  the  13  subject  areas  recognized 
by  the  Armor  branch.9  These  matched  lists  were  to  be  used  during  the  CTSSB  as  the  basis  from 
which  to  extract  two  sets  of  40  critical  tasks  that  would  be  compared  for  site  selection  with  the 
SST’s  embedded  logic,  and  without  the  SST  using  human  judgment  alone.  The  task  lists  were 
created  in  advance  to  save  time  during  the  actual  board  meeting,  recognizing  that  some  of  the 
tasks  on  the  two  sample  lists  could  potentially  be  voted  non-critical  during  the  criticality  voting 
phase.  This  did  indeed  happen,  necessitated  the  addition  of  alternate  tasks  to  the  two  sample 
lists,  as  described  more  fully  below. 


9  The  13  subject  areas  include  (1)  Communications,  (2)  CBRN  (Chemical  Biological  Radiological,  and  Nuclear),  (3) 
Mines,  Minefields,  Demolitions,  (4)  Weapons,  Grenade  launcher.  Anti-tank  weapon,  (5)  Long  Range  Advanced 
Scout  Surveillance  System  (LRAS3),  (6)  Night  Vision  Devices,  (7)  BFV-CFV,  Gunnery,  (8)  Stryker,  HMMWV,  (9) 
M1A1  Tank,  M1A2  SEP,  Gunnery,  (10)  Individual  Skills,  (11)  Urban  Operations,  (12)  Tactics,  Platoon 
Operations/Reports,  and  (13)  Range  Operations,  Logistics. 
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The  board  convened  and  began  their  process,  independent  of  the  SST,  by  validating  the 
LT  and  CPT  task  lists  (indicating  tasks  that  require  modification  or  are  no  longer  valid)  and 
rating  the  criticality  of  each  task  on  a  scale  from  1  to  5.  These  activities  consumed  the  first  three 
days  of  the  CTSSB.  The  CTSSB  detennined  that  all  tasks  with  mean  ratings  of  3.0  or  above 
would  be  considered  critical  tasks. 

The  research  team  then  used  the  task  spreadsheet  produced  by  the  CTSSB,  with  final  task 
criticality  ratings,  to  identify  the  criticality  rating  assigned  to  each  of  the  tasks  on  the  two  80-task 
lists  generated  prior  to  the  board.  Tasks  from  the  two  matched  lists  rated  as  3.5  or  above  were 
highlighted  for  potential  inclusion  in  the  final  lists  of  tasks  to  be  included  in  the  assessment 
study.  One  of  the  task  lists  contained  38  tasks  rated  3.5  or  above.  The  other  list  contained  33 
tasks  rated  3.5  or  above.  These  two  pared  down  lists  were  used  as  the  basis  for  building  Task  Set 
A  (no-SST)  and  Task  Set  B  (SST)  to  be  included  in  the  assessment.  The  lists  were  placed  in  a 
spreadsheet  side  by  side.  Criticality  ratings  were  input  into  the  spreadsheet.  Average  criticality 
ratings  were  calculated  for  each  subject  area,  with  care  taken  to  roughly  match  the  average 
criticality  rating  of  tasks  within  each  subject  area.  The  researchers  and  Armor  SME  reviewed 
these  tasks  and  generated  finalized  matched  Task  Sets  comprised  of  40  tasks  each,  by  adding 
tasks  from  the  master  critical  task  list  as  needed.  Tasks  were  thus  ultimately  matched  for  both 
average  criticality  ratings  across  the  subject  areas,  and  for  task  content.  For  example,  the  task  to 
“mount  a  M240B  machine  gun  on  a  vehicle”  was  considered  to  match  the  task  to  “mount  a 
Caliber  .50  M2  machine  gun  on  a  vehicle.”  As  a  result  of  attrition  due  to  low  criticality  ratings, 
one  of  the  13  subject  areas  was  not  represented  in  the  final  Task  Sets.  Once  the  finalized 
matched  task  lists  of  40  tasks  were  compiled,  they  were  highlighted  in  the  master  19A  critical 
task  list. 

Task  Set  A  (no-SST)  consisted  of  40  critical  tasks  and  was  provided  to  the  CTSSB  for 
human  judgment  via  group  discussion  prior  to  introduction  of  the  SST.  Task  Set  B  (SST) 
consisted  of  an  additional  40  critical  tasks.  Task  Set  B  was  incorporated  as  the  first  40  rows  of 
the  SST,  and  Task  Set  A  was  incorporated  as  the  second  40  rows  of  the  SST.  With  this  design, 
we  were  poised  to  compare  site  recommendations  emerging  from  a  board  discussion  with  those 
produced  by  the  SST’s  logic,  as  well  as  the  tally  of  individual  board  member  inputs. 

SST  Administration  Outcomes.  The  structure  of  the  first  assessment  was  as  follows. 

To  begin  the  period  of  testing  on  Day  4  of  the  CTSSB,  the  research  team’s  Armor  SME  briefed 
the  board  as  to  the  procedure  that  would  be  implemented  regarding  site  selection.  Four  phases 
comprised  the  process.  Phase  I  was  the  conventional  review  of  the  Set  A  tasks,  where  the  board 
convened  to  apply  the  group’s  unaided  judgment  to  place  the  40  tasks  in  training  sites.  In  Phase 
II,  each  board  member  and  the  board  chair  used  the  SST  individually  to  identify  sites  for  Set  B 
followed  by  Set  A  tasks.  In  Phase  III,  the  group  reconvened  to  discuss  the  outcomes  of  the  SST 
for  the  Set  B  tasks,  and  to  settle  on  a  final  site  recommendation  for  the  record.  Finally,  Phase  IV 
involved  administration  of  the  User  Reaction  Survey  to  the  eight  board  members  and  the  board 
chair.  A  focus  group  was  then  conducted  with  the  full  board  and  administrators  who  coordinate 
and  organize  all  the  CTSSBs  for  the  Fort  Benning  site.  Focus  group  topics  included  (1)  usability 
of  the  tool  and  interface;  (2)  relevance  and  clarity  of  the  questions  within  the  SST;  and  (3)  usage 
of  the  SST  in  the  context  of  CTSSBs. 
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In  Phase  I  of  the  testing,  the  board  conducted  its  nonnal  group  site  selection  process  on 
Task  Set  A.  This  involved  identifying  a  location  for  each  task  in  sequence,  with  group 
discussion  of  the  task  as  needed.  In  most  cases  of  the  40,  the  board  chair  identified  the  location. 

In  fewer  than  10  instances,  the  group  discussed  placement.  In  each  of  these  cases,  discussion 
lasted  between  30  seconds  and  two  minutes.  The  factors  considered  in  the  discussion  were 
recorded  by  one  researcher.  In  all,  the  process  of  identifying  training  location  for  the  40  tasks  in 
Set  A  lasted  approximately  fifteen  minutes. 

The  group  then  broke  and  relocated  to  the  computer  lab.  They  were  briefed  on  the  SST, 
including  purpose  of  the  tool  and  instructions  for  its  use.  They  proceeded  to  independently 
review  the  80  tasks  and  identify  a  site  for  each,  requiring  between  35  and  60  minutes  to  complete 
the  SST  process  for  all  80  tasks. 

Researchers  used  the  administrator  compilation  function  to  merge  the  results  of  the  eight 
board  members.  As  a  non-voting  member,  the  board  chair  was  purposely  excluded  from  the 
merged  results.  At  this  stage  of  development,  the  SST  prompted  users  to  simply  agree  or 
disagree  with  the  SST  Recommendation,  in  lieu  of  identifying  their  Board  Member  Opinion  for 
site  selection,  as  is  the  case  in  the  final  version  of  the  tool.  Therefore,  in  the  first  assessment 
study,  the  prototype  compilation  function  automatically  highlighted  individuals’  concurrence 
with  the  SST  site  recommendations  when  the  concurrence  was  70-79%  (amber)  or  less  than  69% 
(red).  In  addition,  researchers  manually  calculated  the  consensus  across  board  members.  A  total 
of  four  calculations  were  generated  by  applying  two  separate  sets  of  consensus  thresholds:  a 
high  (75%  and  62.5%)  and  low  (62.5%  and  50%)  threshold  calculation  for  concurrence 
(agreement  with  the  SST’s  recommendation),  and  a  high  (75%  and  62.5%)  and  low  (62.5%  and 
50%)  threshold  calculation  for  consensus  (agreement  across  board  members’  SST 
recommendations). 

Under  the  first,  higher,  concurrence  threshold,  agreement  was  calculated  across  board 
members  on  SST  recommendations.  Tasks  were  highlighted  amber  when  agreement  was  six  out 
of  eight  (75%)  and  highlighted  red  when  agreement  was  five  or  fewer  out  of  eight  (62.5%  or 
less).  When  agreement  was  seven  or  eight  out  of  eight  (87.5%  or  better),  no  highlights  were 
applied.  As  a  result,  eight  tasks  were  highlighted  amber  and  16  tasks,  red.  Consensus,  adjusted 
for  individual  concurrence  with  the  SST,  was  also  calculated.  When  an  individual  did  not  concur 
with  the  SST,  researchers  assumed  the  individual’s  site  vote  was  for  institution  or  unit,  but  not 
self-development,  given  the  nature  of  the  tasks  under  review. 10  The  same  highlighting  scheme 
was  applied.  Under  this  highlighting  scheme,  13  tasks  were  amber  and  10  were  red. 

Under  the  second,  lower,  concurrence  threshold,  agreement  was  again  calculated  across 
board  members  on  SST  recommendations.  This  time,  tasks  were  highlighted  amber  when 
agreement  was  five  out  of  eight  (62.5%)  and  highlighted  red  when  agreement  was  four  or  fewer 
out  of  eight  (50%  or  less).  When  agreement  was  six  or  greater  out  of  eight  (75%  or  better),  no 
highlights  were  applied.  Twelve  tasks  were  thereby  highlighted  amber,  and  four  tasks,  red.  In 


10  While  this  assumption  proved  accurate  for  the  CTSSB  in  question,  the  need  to  make  such  an  assumption  produced 
a  design  modification,  namely,  for  the  SST  to  prompt  users  for  their  personal  site  recommendation  (i.e.,  Board 
Member  Opinion)  vice  the  simple  agreement  or  disagreement  with  the  SST’s  recommendation,  to  ensure  clarity  as 
to  the  preferred  training  site. 
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addition,  consensus  across  board  members,  adjusted  for  individual  concurrence,  was  likewise 
calculated  for  this  second  threshold,  producing  four  amber  and  five  red  highlighted  tasks. 

The  compiled  recommendations  for  Task  Set  B  were  provided  to  the  board,  with  the  final 
task  highlighting  scheme  applied  (i.e.,  adjusted  for  individual  concurrence,  with  agreement 
<62.5%  highlighted  amber  and  <50%  highlighted  red).  A  total  of  nine  of  the  40  tasks  were 
highlighted  amber  or  red.  The  board  then  discussed  the  placement  of  those  nine  tasks,  since  they 
represented  tasks  where  substantial  disagreement  existed  across  the  board.  One  researcher 
captured  the  rationale  behind  the  placement  of  each  of  those  nine  tasks. 

Assessment  Study  Two 

The  second  round  of  assessment  was  conducted  in  conjunction  with  the  35G  (i.e., 
Geospatial  Imagery  Analyst)  CTSSB  held  at  Fort  Belvoir,  VA.  Due  to  the  classification  of  35G 
tasks  and  the  subsequent  restrictions  on  computer  use  during  the  board,  the  research  team 
examined  the  utility  of  administering  a  paper-based  SST  during  this  CTSSB. 

Participants.  A  total  of  22  individuals  utilized  the  paper-based  SST  during  the  35G 
CTSSB.  These  individuals  included  35G  Staff  Sergeants  (SSGs),  Sergeant  First  Class  (SFC), 
Master  Sergeants  (MSGs),  Warrant  Officers,  and  civilian  training  personnel  (including  the  board 
chair,  training  developer  SMEs,  and  others).  A  total  of  five  voting  board  members  and  10  non¬ 
voting  board  members  completed  the  user  reaction  survey  and  participated  in  the  focus  group. 

Materials.  In  Study  Two,  paper  copies  of  the  flow  chart  diagram,  shown  in  Figure  1, 
above,  replaced  the  computer-based  SST  used  in  Study  One.  In  addition,  the  User  Reaction 
Survey  applied  in  Study  One  was  modified  to  consist  of  12  Likert-scale  queries  rather  than  13. 
All  other  materials  were  identical  to  those  used  in  Study  One. 

Procedure.  The  35G  CTSSB  is  conducted  in  a  secure  setting  and  computer-based 
administration  was  not  deemed  pennissible  as  a  result.  Therefore,  a  paper-based  SST  was 
developed.  The  35G  CTSSB  board  is  run  concurrently  with  the  Warrant  Officer  350G  CTSSB, 
and  some  of  these  personnel  were  present  during  portions  of  the  35G  CTSSB  to  serve  in  an 
advisory  capacity.  The  35G  CTSSB  convened  and  began  its  process,  independent  of  the  SST,  by 
validating,  modifying  when  needed,  and  rating  the  criticality  of  all  35G  tasks  at  all  skill  levels. 
For  the  criticality  vote,  the  board’s  process  included  a  discussion  of  the  task,  its  title  and 
description,  and  a  yes/no  vote  indicating  whether  a  task  should  be  considered  critical.  Consensus 
threshold  was  50%  or  more  (i.e.,  the  majority  rules).  The  board  began  with  lower  level  skills 
(Skill  Level  1/Skill  Level  10:  those  considered  for  Advanced  Individual  Training/AIT)  and 
moved  up  through  Skill  Level  4/Skill  Level  40.  After  each  criticality  vote,  the  board  members 
conducted  a  training  site  vote,  sometimes  but  not  always  after  a  brief  discussion. 

After  completing  the  Skill  Level  2  (Skill  Level  20)  tasks,  participants  were  briefed  on  the 
SST,  including  purpose  of  the  tool  and  instructions  for  its  use.  Participants  were  then 
administered  the  Informed  Consent  and  the  Privacy  Act  Statement.  Beginning  with  Skill  Level  3 
tasks  (also  known  as  Skill  Level  30  tasks,  and  generally  taught  at  ALC/SSG  level),  the  35G 
CTSSB  analyzed  approximately  50  tasks  for  the  ALC  and  approximately  10  Skill  Level  4  (also 
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known  as  Skill  Level  40,  SLC/SFC  level)  critical  tasks  for  site  recommendations  using  the 
paper-based  SST.  Applying  an  earlier  version  of  the  decision  tree  depicted  in  Figure  1, 
participants  continued  voting  on  the  criticality  of  each  remaining  task  and  modifying  the  task  as 
needed.  After  finalizing  the  task  title  and  description  and  conducting  the  criticality  vote,  board 
members  silently  and  individually  worked  through  the  paper-based  SST.  Beginning  with 
Question  1  in  the  flow  chart  in  Figure  3,  participants  answered  each  question  with  the  same 
response  options  as  the  computer-based  SST.  Each  task’s  site  detennination  took  approximately 
1  minute. 

After  each  round  of  site  determination  (for  each  task),  the  board  chair  asked  each  board 
member  to  report  aloud  the  training  site  that  the  paper-based  SST  recommended.  Then,  the  chair 
solicited  the  board  member’s  personal  training  site  recommendation.  When  agreement  was  not 
reached,  the  board  discussed  the  task  and  reached  a  group  consensus  (although  not  always  by 
conducting  an  additional  vote),  and  the  board  administrator  recorded  the  final  training  site 
decision.  Due  to  the  sensitive  nature  of  the  tasks  and  restrictions  on  note-taking  in  this 
environment,  the  research  team  did  not  record  the  specific  training  sites  recommended  by  the 
paper-based  SST,  by  the  board  members,  or  the  final  training  site  recorded.  Rather,  Assessment 
Study  Two  observations  were  focused  on  the  process  and  feasibility  of  utilizing  a  paper-based 
SST,  and  the  board  members’  feedback  regarding  the  logic  guiding  the  decision  tree. 

At  the  conclusion  of  the  criticality  and  site  detennination  votes  for  Skill  Level  4  tasks,  the 
research  team  administered  the  User  Reaction  Survey  to  the  voting  board  members,  non-voting 
board  members,  and  other  training  personnel  present.  Voting  board  member  surveys  were  held 
separate  from  the  non-voting  board  member  and  advisory  personnel  surveys.  After  completion 
of  the  survey,  one  research  psychologist  conducted  an  abbreviated  focus  group  using  portions  of 
the  focus  group  protocol.  Lines  of  questioning  focused  on  understanding  how  accurate  the  tool’s 
recommendations  were  and  where  editing,  removing,  or  adding  questions  could  improve  the 
decision  tree  guiding  the  tool. 

Two  notable  departures  from  the  computer-based  SST  existed.  The  first  was  the  lack  of 
an  administrator  version  to  notate  separately  the  SST  concurrence  and  the  board  member  opinion 
consensus.  However,  this  process  was  somewhat  paralleled  by  the  chair,  who  elicited  personal 
opinion  recommendations  in  addition  to  the  SST  recommendation,  and  capitalized  on 
discrepancies  between  the  two  opinions  as  a  catalyst  for  discussion.  In  keeping  with  the  general 
SST  process,  items  of  high  disagreement  were  discussed,  and  the  final  CTSSB  recorded 
recommendation  resulted  from  this  discussion. 

The  second  notable  departure  from  the  computer-based  SST  was  the  voting  process. 
Although  board  members  generally  worked  independently  while  completing  the  paper-based 
SST,  reporting  their  votes  aloud  in  sequence  may  have  caused  board  members  to  reconsider  their 
responses  while  listening  to  others,  perhaps  changing  their  responses  as  a  result  of  listening  to 
the  rationale  of  other  board  members.  Likewise,  the  board  members’  personal  opinions  may 
have  also  been  informed  by  listening  to  other  board  members’  rationale  during  the  personal 
opinion  reporting  process.  This  potential  to  preemptively  influence  votes  before  a  discussion 
dilutes  the  effect  of  the  SST’s  ability  to  tap  into  the  varied  opinions  of  board  members.  The 
process  issues  identified  via  this  CTSSB  experience  were  addressed  in  the  final  paper-based  SST 
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instructions,  by  suggesting  adaptations  to  reduce  the  potential  influence  on  personal  votes,  and 
delaying  the  influence  of  others’  until  the  group  discussion. 

Assessment  Study  Three 

The  final  round  of  assessment  was  conducted  during  an  NCO  CTSSB  at  Fort  Huachuca, 
AZ,  covering  the  35T  MOS  (Military  Intelligence  Systems  Maintainer/Integrator).  In  this  study, 
the  entire  critical  task  list  was  subjected  to  site  recommendations  via  the  SST. 

This  assessment  study  was  designed  to  compare  SST  location  recommendations  to 
human  location  recommendations,  and  elicit  user  feedback.  User  reaction  surveys  were  collected 
from  CTSSB  board  members  and  chairs,  and  feedback  was  collected  from  board  members,  board 
chairs,  and  administrators  in  the  context  of  a  post- test  focus  group. 

Participants.  A  total  of  17  participants  completed  the  computer-based  SST  and  the  User 
Reaction  Survey,  and  participated  in  the  focus  group.  All  participants  were  voting  CTSSB 
members  with  ranks  of  SSG,  SFC,  or  MSG. 

Materials.  In  Study  Three,  the  computer-based  SST  was  applied  just  as  in  Study  One. 

All  other  materials  were  identical  to  those  employed  in  Study  Two. 

Procedure.  The  35T  CTSSB  board  is  run  concurrently  with  the  Warrant  Officer  350T 
CTSSB,  and  some  of  these  personnel  were  present  during  portions  of  the  35T  CTSSB  to  serve  in 
an  advisory  capacity.  The  35T  CTSSB  convened  and  began  its  process,  independent  of  the  SST, 
by  validating,  modifying  when  needed,  and  rating  the  criticality  of  all  35T  tasks  at  all  skill  levels. 
For  the  criticality  vote,  the  board’s  process  included  a  discussion  of  the  task,  its  title  and 
description,  and  a  yes/no  vote  indicating  whether  a  task  should  be  considered  critical.  Consensus 
threshold  was  50%  or  more  (i.e.,  majority  rules).  The  board  began  with  lower  level  skills  (Skill 
Level  1/Skill  Level  10:  those  considered  for  AIT)  and  progressed  through  Skill  Level  4/Skill 
Level  40.  Like  the  19A  CTSSB  but  unlike  the  35G  CTSSB,  the  training  site  detennination 
process  was  conducted  as  a  separate  phase  following  finalization  of  the  critical  task  list. 

This  board  was  unique  in  that  it  represented  a  particularly  heterogeneous  group  of  three 
recently  merged  MOSs.  As  such,  board  members  possessed  considerably  distinct  military 
experiences,  and  were  not  familiar  with  every  task  in  the  total  task  inventory.  Therefore,  the 
board  chair  subdivided  the  group  into  separate  working  groups  reflecting  the  unique  areas  of 
expertise.  These  working  groups  assessed  the  criticality  of  the  tasks  within  their  area  of 
expertise  and  through  discussion,  decided  upon  the  criticality  of  the  tasks.  These  criticality 
recommendations  were  then  passed  to  the  larger  CTSSB,  which  generally  adopted  them  without 
modification. 

After  completing  the  criticality  vote  of  all  skill  level  tasks,  the  finalized  critical  task  list 
was  input  into  the  computer-based  SST,  and  the  file  was  distributed  to  computers  in  an  Army 
computer  lab.  Participants  were  briefed  on  the  SST,  including  purpose  of  the  tool  and 
instructions  for  its  use.  Participants  were  then  administered  the  Informed  Consent  and  the 
Privacy  Act  Statement.  They  proceeded  to  independently  review  the  67  critical  tasks  and 
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identify  a  site  for  each,  requiring  approximately  60  minutes  to  complete  the  SST  process  for  all 
tasks. 


Researchers  applied  the  administrator  compilation  function  to  merge  the  results  of  the  17 
board  members.  The  prototype  compilation  function  automatically  highlighted  tasks  red  when 
consensus  was  not  achieved  for  one  of  two  measures:  (1)  SST  recommendations  did  not  attain  a 
65%  consensus  and  board  members  opinion  did  not  attain  65%  consensus. 

The  compiled  recommendations  with  the  highlighted  tasks  were  provided  to  the  board.  A 
total  of  54  of  the  67  tasks  were  highlighted  as  a  function  of  one  of  the  two  flagging  measures:  47 
tasks  were  flagged  as  a  result  of  low  SST  consensus  and  42  tasks  were  flagged  as  a  result  of  low 
board  member  opinion  consensus.  The  board  then  re -voted  or  discussed  the  placement  of  those 
flagged  tasks,  since  they  represented  tasks  where  substantial  disagreement  existed  across  the 
board. 


Due  to  time  constraints,  before  the  CTSSB  completed  its  discussion  of  the  flagged  tasks, 
the  research  team  asked  the  voting  board  members  to  complete  the  User  Reaction  Survey.  After 
completion  of  the  survey,  one  research  psychologist  conducted  an  abbreviated  focus  group  using 
portions  of  the  focus  group  protocol.  Participants  received  a  copy  of  the  decision  tree  from 
Figure  1  to  illustrate  the  logic  guiding  the  SST.  Lines  of  questioning  focused  on  understanding 
how  accurate  the  tool’s  recommendations  were  and  where  the  decision  tree  guiding  the  tool 
could  be  improved  by  editing,  removing,  or  adding  questions. 

Results 

Site  selection  data  and  user  reactions  were  examined  for  the  three  studies.  Each 
assessment  site  posed  a  unique  challenge.  The  19A  board  reviewed  LT  vice  NCO  tasks,  and  thus 
the  majority  of  tasks  required  introductory  training  in  the  ABOLC  (i.e.,  institutional)  setting.  The 
35G  board  conducted  a  paper-based  site  selection  methodology  using  only  a  small  number  of 
tasks.  Finally,  the  35T  board  reviewed  tasks  for  a  single  MOS  recently  merged  from  three 
separate  MOSs,  and  board  members  were  only  able  to  apply  personal  experience  to  one  of  those 
three  MOSs.  With  these  challenges  in  mind,  the  research  team  relied  heavily  on  user  comments 
and  suggestions,  as  well  as  feedback  from  experienced  CTSSB  administrators,  to  modify  the 
tool’s  functionality  following  each  round  of  assessment. 

Comparison  of  SST  Recommendations  to  Board  Member  Opinions  and  Final  Board 
Recommendations 

Fort  Benning  19A  CTSSB.  Task  Set  A  -  the  list  of  40  tasks  assessed  for  site  placement 
prior  to  application  of  the  SST,  and  then  again  with  the  SST  -  was  subjected  to  a  comparison  of 
SST  recommendations  with  board  member  opinions.  The  board  decided  to  change  the  final  site 
recommendation  as  a  result  of  the  application  of  the  SST  in  three  out  of  the  1 1  instances  from 
Task  Set  A  when  the  SST  recommendation  differed  from  the  initial,  pre-SST  board  vote.  This 
finding  suggests  that  following  use  of  the  SST,  board  members  adjusted  their  site  selection 
considerations.  In  seven  of  these  1 1  cases,  the  final  board  decision  did  not  match  the  tally  of 
individual  board  member  opinions  as  recorded  within  the  SST.  This  finding  suggests  that  the 
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SST  may  be  more  successful  than  the  current  processes  at  revealing  differences  of  opinion 
between  individual  board  members. 

Within  Task  Set  A  alone,  when  the  SST  recommended  the  institution  as  the  training  site, 
individual  board  members  agreed  with  the  placement  83%  of  the  time.  However,  when  the  SST 
recommended  the  unit  as  the  training  site,  agreement  decreased  to  only  34%.  Within  Task  Set  B 
alone,  individuals  agreed  with  the  SST  recommendation  of  institution  95%  of  the  time  and  with 
the  SST  recommendation  of  unit  training  40%  of  the  time.  Table  5  presents  these  results.  The 
relatively  high  agreement  with  a  vote  for  the  institution  and  relatively  low  agreement  with  a  vote 
for  the  unit  is  likely  related  to  the  perception  that  these  LT  tasks  must  be  at  least  introduced  in  an 
institutional  setting  prior  to  the  young  officer  reporting  to  duty  at  a  unit.  The  tool  did  not 
recommend  self-development  for  any  of  the  tasks  in  Set  A  or  Set  B. 

Agreement  with  the  SST  recommendations  was  also  calculated  by  summing  the  board 
member  results  as  a  group.  These  results  are  also  presented  in  Table  5.  Within  Task  Set  A, 
when  comparing  group  consensus  on  the  SST  recommendations  to  group  consensus  on  board 
members’  personal  site  placement  opinions  at  the  75%  level  (or  six  out  of  eight  respondents), 
board  members  agreed  with  the  SST  recommendation  50%  of  the  time.  When  the  same 
agreement  was  measured  at  the  62.5%  level  (or  five  out  of  eight  respondents),  participants 
agreed  with  the  SST  80%  of  the  time. 

Table  5 


19A  CTSSB  Individual  and  Group  Agreement  with  SST  Recommendations 


19A  CTSSB 

19A  CTSSB 

Task  Set  A 

Task  Set  B 

Individual  Agreement  with  SST 
Recommendation  of  Institution 

83% 

95% 

Individual  Agreement  with  SST 
Recommendation  of  Unit 

34% 

40% 

Collective  SST  Recommendation  and 

Collective  Board  Member  Opinion 
(75%  threshold) 

50% 

(20/40) 

57% 

(23/40) 

Collective  SST  Recommendation  and 

Collective  Board  Member  Opinion 
(62.5%  threshold) 

80% 

(32/40) 

82.5% 

(33/40) 

Collective  SST  Recommendation  and  Final 
Group  Decision 
(75%  threshold) 

52.5% 

(21/40) 

55% 

(22/40) 

Collective  SST  Recommendation  and  Final 
Group  Decision 
(62.5%  threshold) 

77.5% 

(31/40) 

75% 

(30/40) 

Individual  Board  Members’  Opinion  and 

75% 

80% 

Final  Group  Decision  (75%  threshold) 

(30/40) 

(32/40) 

Individual  Board  Members’  Opinion  and 

75% 

90% 

Final  Group  Decision  (62.5%  threshold) 

(30/40) 

(36/40) 
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SST  recommendations  were  likewise  compared  to  the  board’s  final  site  selection 
recommendation.  When  measuring  agreement  within  Task  Set  A  at  the  75%  level,  or  six  out  of 
eight  respondents,  the  SST  recommendation  matched  the  final  board  recommendation  52.5%  of 
the  time.  Using  a  62.5%  threshold  for  measuring  agreement,  the  SST  agreed  with  the  final  board 
recommendation  77.5%  of  the  time.  Although  there  is  not  perfect  agreement  between  the  SST 
and  the  final  board  recommendation,  the  levels  indicate  that  the  SST  captures  similar  factors  and 
reasoning  as  those  occurring  within  group  discussion.  Moreover,  in  Task  Set  B,  when  the  group 
discussed  and  formally  voted  on  tasks  after  they  were  subjected  to  the  SST,  the  SST  agreed  with 
the  final  group  decision  more  often  than  not.  Regardless,  utilizing  the  SST  requires  that  board 
members  consider  many  aspects  of  critical  tasks  that  they  may  not  automatically  consider. 
Therefore,  the  quality  of  the  responses  after  utilizing  the  SST,  whether  or  not  they  agree  with  the 
final  group  decision,  should  be  more  reasoned.  The  value  of  the  SST,  even  after  group 
discussion,  is  evident  in  the  19A  board  in  that  after  completing  the  SST  on  Task  Set  A,  the  board 
recognized  additional  factors  deserving  of  consideration,  and  as  such,  revisited  some  of  these 
tasks  and  revised  their  final  group  decision. 

Finally,  individual  board  member  opinions  on  Task  Set  A  as  reported  in  the  SST  were 
compared  with  the  final  board  recommendations.  The  individual  board  member  votes  were  in 
agreement  with  the  final  board  decisions  75%  of  the  time  under  both  the  75%  and  62.5%  levels. 

When  agreement  between  the  SST  and  board  members  as  a  group  was  examined  for  Task 
Set  B,  the  SST  and  board  member  opinion  votes  were  in  agreement  57%  of  the  time  at  the  75% 
level  (six  out  of  eight  participants)  and  82.5%  of  the  time  at  the  62.5%  level  (five  out  of  eight 
respondents).  The  SST  recommendation  matched  the  final  group  recommendation  55%  of  the 
time  at  the  75%  level  (six  out  of  eight)  and  75%  of  the  time  at  the  62.5%  level  (five  out  of  eight 
votes).  Finally,  the  individual  board  member  votes  were  in  agreement  with  the  final  board 
decisions  80%  of  the  time  under  the  75%  level,  and  90%  of  the  time  at  the  62.5%  level.  These 
results  reflect  the  bias  of  the  19A  CTSSB  toward  directing  entry-level  LT  tasks  to  the 
institutional  setting,  even  if  the  task  characteristics  suggest  otherwise.  This  practice  ensures  that 
all  incoming  Annor  officers  receive  introductory  training  across  the  preponderance  of  LT  tasks 
prior  to  their  first  unit  assignment.  Interestingly,  the  results  also  reveal  a  degree  of  variability  in 
board  member  opinions  that  may  be  unanticipated  by  CTSSB  administrators  and  course 
managers. 

Fort  Belvoir  35G  CTSSB.  Due  to  the  sensitive  nature  of  the  tasks  and  restrictions  on 
note-taking  in  this  environment,  the  research  team  did  not  record  the  specific  training  sites 
recommended  by  the  paper-based  SST,  by  the  board  members,  or  the  final  training  site  recorded. 
Rather,  Assessment  Study  Two  observations  were  focused  on  the  process  and  feasibility  of 
utilizing  a  paper-based  SST,  and  the  board  members’  feedback  regarding  the  logic  guiding  the 
decision  tree. 

Fort  Huachuca  35T  CTSSB.  The  entire  critical  task  list  of  67  tasks  was  submitted  to 
the  SST  for  site  selection.  As  a  result  of  modifications  to  the  tooFs  functionality  following  early 
rounds  of  assessment,  data  regarding  individual  board  member  agreement  with  the  SST 
recommendation  was  not  readily  available  in  the  compiled  administrator  SST.  However,  we  can 
infer  and  approximate  such  agreement  by  summing  the  total  SST  site  recommendations  for  each 
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task  and  comparing  it  to  the  summed  totals  of  board  member  site  votes  for  each  task.  With  this 
calculation,  board  members  agreed  with  the  SST’s  recommendation  of  “institution” 
approximately  88.9%  of  the  time,  with  “unit”  approximately  97.6%  of  the  time,  and  with  “self- 
study”  an  estimated  86%  of  the  time.  These  findings  suggest  the  SST  produces  reliable  site 
recommendations. 

In  stark  contrast,  the  board  as  a  group  did  not  exhibit  high  levels  of  consensus  with  each 
other  on  either  the  SST  recommendation  nor  board  member  opinions.  In  47  cases  out  of  66 
(71.2%),  board  member  consensus  on  the  SST  recommendation  was  equal  to  or  less  than  the 
65%  threshold.  In  other  words,  12  or  more  out  of  the  17  board  members  concurred  with  each 
other’s  SST  recommendation  on  about  29%  of  the  tasks.  Similarly,  board  member  site 
placement  opinions  lacked  greater  than  65%  consensus  for  42  tasks  out  of  66  (63.6%),  meaning 
that  12  or  more  of  the  17  board  members  were  in  concurrence  on  personal  site  placement 
opinions  only  about  36%  of  the  time.  These  findings  highlight  a  lack  of  consistent  judgment 
among  board  members  with  or  without  implementation  of  the  SST,  most  likely  due  to  the 
representation  across  three  disparate  MOSs.  In  fact,  many  board  members  reported  informally 
that  their  insufficient  expertise  on  many  of  the  tasks  hindered  their  ability  to  make  confident  site 
recommendations  of  their  own.  As  a  result,  their  less-infonned  responses  introduced  variation 
and  noise  into  the  participants’  responses,  weakening  the  group’s  ability  to  reach  consensus. 
However,  these  findings  also  demonstrate  that  SST  recommendations  were  trusted  as  reasonable 
when  a  lack  of  experience  precluded  a  strong  personal  opinion. 

User  Reaction  Surveys.  Mean  ratings  were  calculated  for  the  survey  questions  for  each 
of  the  three  rounds  of  assessment,  and  appear  in  Table  6.  While  the  ratings  from  the  35T  CTSSB 
were  generally  lower  than  ratings  at  the  other  two  sites,  this  difference  can  be  attributed  to  the 
challenge  of  assessing  tasks  across  three  MOSs  when  each  board  member  was  able  to  represent 
only  one.  In  fact,  eight  respondents  noted  the  SST  should  be  changed  to  allow  for  board 
members  to  opt  out  of  responding  if  they  lack  experience  in  the  task.  This  modification  was 
made  to  the  SST,  so  that  board  members  could  skip  a  task  in  the  list  and  the  Administrator’s 
compilation  would  reflect  a  fewer  number  of  votes. 

In  general,  the  respondents  found  the  SST  questions  easy  to  understand  and  the  tool 
simple  to  use.  Moderate  agreement  was  found  with  statements  regarding  the  SST’s  value  and 
impact  on  board  members’  consideration  of  a  broader  range  of  site  selection  factors.  Of  interest 
is  the  finding  that  user  ratings  were  generally  higher  for  use  of  the  paper-based  site  selection 
methodology  (in  the  35G  CTSSB)  than  for  the  computer-based  SST.  These  differences  may  be 
attributable  to  the  fact  that  apart  from  the  classification  issues  precluding  use  of  the  computer 
lab,  the  SST  was  designed  for  an  NCO  board  typified  by  the  35G  board.  It  was  not  designed  for 
introductory  LT  tasks,  and  it  was  not  prepared  for  a  situation  where  most  board  members  were 
unfamiliar  with  the  tasks  under  review.  Lower  ratings  for  the  19A  board  may  also  reflect  use  of 
an  earlier,  less  refined  version  of  the  SST.  The  lower  ratings  by  the  35T  board  may  reflect  the 
fact  that  these  board  members  lacked  a  basis  of  comparison  between  site  determination  with  and 
without  the  SST.  For  members  having  no  exposure  to  the  obstacles  associated  with  free-form 
group  discussion  of  optimal  training  sites,  it  may  be  impossible  to  judge  the  advantages  of  the 
SST  for  site  detennination. 
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Table  6 


Mean  Ratings  on  User  Reaction  Survey  where  1  =  Strongly  Disagree  and  5  =  Strongly  Agree 


Assessment  Survey  Question 

19A  CTSSB 
Average 
(n=9) 

35G  CTSSB 
Average 
(n=15) 

35T  CTSSB 
Average 
(n=17) 

1.  The  SST  was  valuable  to  me. 

3.13 

3.13 

2.82 

2.  The  questions  in  the  SST  were  easy  to  understand. 

4.33 

4.33 

2.53 

3.  The  questions  in  the  SST  were  easy  to  answer. 

3.87 

3.87 

2.77 

4.  The  SST  was  simple  to  use.11 

4.00 

3.40 

2.82 

5.  I  was  able  to  be  more  objective  with  my  site 
recommendations  using  the  SST. 

3.00 

6.  I  was  able  to  produce  better  site  recommendations 
using  the  SST. 

2.78 

3.29 

2.77 

7.  The  SST  helped  me  to  consider  factors  1  would  not 
have  otherwise  thought  about  when  recommending  a 
training  site. 

3.11 

4.00 

3.12 

8.  The  training  site  recommendation  generated  by  the 

SST  typically  met  my  expectations. 

2.44 

3.00 

2.53 

9.  The  SST  is  reliable. 

2.56 

3.29 

2.47 

10.  The  SST  was  accurate  in  recommending  the  correct 
site  for  each  task. 

2.67 

3.00 

2.29 

11.  Using  the  SST  was  NOT  time  consuming. 

3.11 

4.40 

2.41 

12.  I  would  recommend  the  SST  for  future  CTSSBs. 

2.44 

3.77 

2.94 

13.  All  components  of  the  SST  functioned  properly. 

3.44 

3.33 

3.88 

Discussion  of  User  Feedback 

Overall  verbal  feedback  on  the  SST  was  favorable.  Notably,  individuals  who  administer 
or  are  otherwise  involved  in  CTSSBs  on  a  regular  basis  reported  more  favorable  opinions  of  the 
tool  than  board  members,  who  have  never  before  seen  a  CTSSB.  Since  the  non-board  members 
have  a  better  basis  of  comparison  with  which  to  judge  the  value  added  and  gaps  fdled  by  the 
SST,  the  research  team  lent  more  credence  to  their  substantive  comments  than  the  board 
members. 

Nonetheless,  board  member  users  did  note  that  the  SST  functions  as  is  intended  -  it 
results  in  them  applying  more  thought  and  considering  a  broader  range  of  factors  when 
considering  task  placement,  and  it  flags  for  group  discussion  those  tasks  that  require  a  more  in- 
depth  dialogue  about  the  optimal  site  placement.  They  saw  value  in  the  tool  as  a  systematic 
process  that  enables  the  human  to  apply  thought,  and  the  system  to  analyze  those  judgments  to 
produce  a  recommendation.  They  also  noted  the  SST  to  be  a  better  means  of  producing  group 
discussion  than  the  current  practice,  where  no  precursor  exists  at  all  to  prompt  discussion  for  a 
particular  task. 


11  Questions  4  and  5  were  combined  into  a  single  question  during  the  35G  and  35T  administrations. 
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Implementation  of  SST  in  the  CTSSB  Context.  Although  the  SST  was  conceptualized 
as  a  tool  to  augment  board  member  judgment,  users  often  expressed  a  concern  that  the  Army’s 
intent  may  be  otherwise.  With  current  Anny  practices  to  increase  the  efficiency  of  various 
processes  through  online  surveys  and  other  mechanisms  by  which  individual  inputs  are  captured 
asynchronously,  many  feared  the  tool  would  be  eventually  posted  online,  with  the  requirement 
for  board  members  to  submit  their  she  selections  prior  to  convening  of  the  board.  This  course  of 
action  was  strongly  discouraged. 

Board  members  stated  that  the  SST  provides  a  “warm  up”  for  contemplating  the  factors 
to  be  considered  when  recommending  she  placement.  In  a  few  cases,  users  recommended  the 
SST  be  applied  on  Day  1  of  the  CTSSB  with  a  sampling  of  the  tasks,  in  order  to  organize  and 
facilitate  the  task  discussions  that  would  occur  over  the  course  of  the  CTSSB  week.  A  few 
individuals  suggested  the  SST  be  applied  prior  to  arrival  at  the  CTSSB  so  that  individual 
opinions  and  responses  could  be  viewed  without  contamination  from  the  group-think 
phenomenon  or  the  influence  of  strong  personalities  driving  the  discussion.  These  suggestions 
were  discarded  due  to  the  requirement  to  apply  the  SST  only  to  the  critical  task  list,  which  does 
not  take  shape  until  midway  through  the  CTSSB.  Moreover,  utilizing  the  tool  prior  to  the 
identification  of  critical  tasks  would  also  increase  the  time  required  for  completion  as  all  tasks 
would  be  assessed,  not  just  the  shorter  critical  task  list.  For  some  CTSSBs,  this  would  be  a 
substantially  large  number.  Finally,  from  a  purely  practical  standpoint,  board  members  have 
limited  preparation  time  between  notification  of  their  participation  in  the  impending  board  and 
the  boards’  commencement,  which  could  preclude  implementation  of  the  SST  prior  to  arrival  at 
the  CTSSB. 

Indicating  High  versus  Low  Consensus.  The  compilation  of  board  member  site 
recommendations  in  Studies  One  and  Three  illustrated  a  need  to  modify  the  means  by  which  the 
SST  highlights  tasks  for  which  consensus  is  low.  In  Study  One,  when  LT  tasks  populated  the 
SST,  consensus  improved  when  board  member  opinions  were  compared  to  SST 
recommendations.  The  primary  driver  for  the  difference  between  personal  opinion  and  SST 
recommendation  related  to  the  fact  that  LT  is  the  entry  level  rank  in  the  officer  corps.  That  is, 
because  the  ABOLC  -  the  institutional  POI  under  review  -  would  be  the  LTs’  first  exposure  to 
all  the  tasks  on  the  list,  it  was  deemed  critical  to  address  a  substantial  number  of  them  in  the 
institutional  setting  at  least  to  an  introductory  level.  In  Study  Three,  the  board  was  subject  to  a 
special  circumstance  in  which  three  MOSs  were  undergoing  a  merger  into  one,  every  board 
member  represented  one  of  the  MOSs,  and  therefore  for  every  task,  several  board  members  were 
inexperienced  in  its  conduct.  As  a  result,  consensus  tended  to  be  relatively  low  across  the  entire 
task  set.  In  this  board,  the  chair  determined  that  a  majority  vote  (i.e.,  >50%)  was  sufficient  as  a 
threshold.  Because  other  CTSSBs  will  similarly  experience  special  circumstances,  whether  it  be 
having  only  a  few  board  members,  task  lists  that  are  outliers,  or  other  task  modifications  within 
the  MOS,  the  detennination  was  made  to  provide  customizable  consensus  thresholds.  In  the  final 
version  of  the  SST,  the  administrator  may  select  from  50%,  60%,  70%,  or  80%  as  the  threshold 
applied  by  the  SST.  The  decision  to  implement  a  higher  or  lower  threshold  is  at  the  discretion  of 
the  board  administrator  and/or  board  chair,  as  he/she  sees  fit.  Furthermore,  since  user  groups 
were  not  discriminating  the  red  from  amber  highlights  in  their  discussions,  only  a  single 
highlight  color  (amber)  is  applied  in  the  final  version  of  the  tool. 
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NCOES  versus  Functional  Courses.  In  every  CTSSB,  site  selection  is  conducted 
considering  one  particular  course  as  the  “Institution.”  The  courses  within  the  NCOES  and 
therefore  the  focus  of  this  study  were  WLC,  and  each  MOS’s  Advanced  Leader  Course  (ALC) 
and  Senior  Leader  Course  (SLC).  However,  functional  courses  are  also  provided  by  the 
institutions,  typically  to  provide  for  technical  training  on  a  particular  system,  vehicle,  piece  of 
equipment,  or  skill.  Therefore,  board  members  require  an  option  to  suggest  a  functional 
institutional  course  as  the  optimal  site,  vice  the  course  under  review  as  the  institutional  site.  In 
early  iterations  of  the  SST,  users  were  instructed  to  use  the  Comment  column  to  identify  tasks 
recommended  for  the  Institution,  but  better  suited  to  a  functional  course  rather  than  NCOES 
course.  However,  the  final  version  of  the  tool  was  modified  to  provide  users  the  response  option 
“Institutional-Functional”  when  asked  where  they  would  recommend  the  task  be  trained  if  they 
disagree  with  the  SST’s  recommendation.  The  SST  itself  does  not  recommend  functional 
courses  as  a  training  site. 

Gaming  the  System.  Many  individuals  noted  their  propensity  to  “game  the  system,” 
meaning  that  they  attempted  to  figure  out  the  algorithms  so  that  they  could  answer  the  questions 
to  produce  their  desired  site  recommendation.  To  better  control  against  gaming  behaviors,  the 
research  team  built  in  a  means  of  reporting  one’s  personal  opinion  in  a  way  that  was  weighted 
equally  to  the  SST  recommendation.  However,  this  modification  did  not  seem  to  decrease  the 
gaming  behaviors.  We  considered  displaying  the  SST  recommendations  only  after  all  questions 
were  answered  for  every  task,  and  the  user  clicked  a  button  to  submit  the  answers.  However,  this 
approach  was  jettisoned  due  to  the  additional  time  requirement  it  would  have  produced.  We 
deemed  it  crucial  to  collect  user  opinions  in  addition  to  SST  recommendations;  modifying  the 
functionality  as  described  would  require  users  to  review  the  task  list  a  second  time  to  examine 
the  SST  recommendation  and  respond  as  to  its  agreement  with  their  own  personal  opinion. 

Availability  of  Equipment  at  the  Institution.  While  the  front-end  analysis 
methodology  and  SST  include  a  question  as  to  whether  units  are  likely  to  have  the  equipment 
required  to  train  the  task,  a  similar  question  regarding  equipment  available  at  the  institution  was 
purposefully  withheld  from  the  methodology  and  SST.  However,  board  members  who  tested  the 
SST  suggested  the  tool  should  in  fact  include  a  question  as  to  equipment  availability  at  the 
institution.  This  suggestion  was  not  implemented.  The  SMEs  in  the  initial  interviews  described 
that  course  managers,  developers,  and  instructors  review  the  CTSSB  results,  and  apply  an 
additional  level  of  judgment  as  to  site  placement  recommendations  based  in  part  upon  the 
institution’s  resources  (to  include  time,  funding,  and  equipment).  When  institutional  resourcing 
cannot  support  the  CTSSB’s  recommendations,  the  subsequent  recommendations  to  the  NCO 
Academy  Commandant  are  adjusted  accordingly.  Therefore,  any  judgment  regarding 
institutional  resources  is  applied  outside  the  CTSSB  process,  since  board  members  are  unable  to 
accurately  assess  the  resources  of  the  institution.  In  contrast,  however,  board  members  would 
generally  have  knowledge  of  a  unit’s  access  to  equipment  necessary  for  training,  so  they  are 
capable  of  answering  that  question  within  the  SST. 

Subjectivity  of  Judgment.  A  few  users  noted  that  responses  to  the  questions  can  be 
subjective,  and  viewed  that  as  a  negative  trait  of  the  SST.  However,  site  selection  by  its  nature 
cannot  be  a  purely  objective  undertaking.  The  current  site  selection  process  is  wholly  subjective 
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with  no  guidelines  for  decision  making.  The  SST  provides  a  set  of  guidelines,  similar  to  the  task 
criticality  ratings,  within  which  to  make  a  subjective  recommendation. 

Responding  to  All  Principal  Discriminators.  A  subset  of  users  indicated  a  desire  to 
answer  all  twelve  questions  for  every  task.  They  felt  that  some  of  the  questions  grayed  out  as  a 
result  of  the  branching  logic  were  actually  important  questions.  However,  they  simultaneously 
noted  that  they  did  not  disagree  with  the  SST’s  site  recommendation  in  these  instances,  which 
means  that  answering  an  additional  question  would  have  no  bearing  on  the  final  SST 
recommendation.  The  tool  was  designed  to  minimize  the  number  of  responses  required  to 
produce  a  sound  recommendation,  because  in  many  cases  the  critical  task  list  will  number  more 
than  100  tasks.  The  19A  task  list  for  LTs  consisted  of  more  than  550  tasks.  Therefore,  the  team 
did  not  modify  the  tool  to  allow  users  to  answer  all  twelve  questions  for  every  task. 

Discussion  and  Conclusions 

Early  indications  suggest  that  the  SST  is  indeed  useful  to  support,  but  not  replace,  the 
human  judgment  applied  by  CTSSB  board  members  for  their  site  selection  recommendations. 
While  the  Army  moves  toward  the  implementation  of  the  Army  Learning  Model,  which  in  part 
prescribes  better  use  of  technologies  to  support  training  requirements,  the  SST  provides  the  force 
with  a  tool  to  make  informed  decisions  as  to  tasks  that  can  be  trained  via  technology  (i.e., 
structured  self-development  modules)  versus  those  for  which  face-to-face  classroom  instruction 
with  a  skilled  instructor  is  still  preferable. 

The  SST  is  intended  to  aid  the  Anny  at  large  in  making  better  training  site  detennination 
decisions.  This  is  accomplished  in  several  ways.  First,  the  SST  encapsulates  the  expertise  of 
diverse  site-selection  SMEs  into  a  process  that  informs  and  educates  board  members  on  factors 
they  should  consider  when  making  site-detennination  decisions.  This  education  process 
improves  the  CTSSB  board  member’s  decision-making.  However,  it  is  important  to  reiterate 
that  the  SST  does  not  mandate  that  each  CTSSB  follows  the  SST  recommendation;  every 
CTSSB  has  unique  concerns  and  issues  which  may  not  be  fully  captured  by  the  SME  expertise 
represented  in  the  SST.  Instead,  board  members  are  free  (and  solicited)  to  express  disagreement 
with  the  SST  if  they  do  in  fact  disagree.  Second,  some  CTSSBs  operate  without  extensive 
discussion  or  a  formal  vote  (e.g.,  one  or  two  members  nominating  a  training  site,  which  is 
confirmed  if  there  are  no  objections).  In  these  situations,  the  SST  highlights  areas  of 
disagreement  that  may  be  overlooked  by  ensuring  that  minority  opinions  are  recorded  and  less 
assertive  individuals  may  be  called  to  explain  a  valuable  divergent  opinion.  In  contrast,  a  third 
way  the  SST  aids  better  training  site  decision  making  is  in  identifying  the  tasks  that  have  high 
levels  of  existing  agreement.  In  this  way,  CTSSBs  do  not  spend  valuable  time  discussing  the 
minutiae  of  tasks  upon  which  board  members  disagree  when  in  fact  board  members  agree  on  the 
broader  issue  of  training  site  recommendation.  Instead,  the  SST  identifies  tasks  that  already  have 
high  levels  of  agreement  and  need  not  be  discussed,  reserving  valuable  meeting  time  for  the  tasks 
upon  which  there  is  greater  discrepancy  and  divergence  of  opinions. 

However,  it  is  critical  that  the  users  view  and  use  SST  outcomes  appropriately  and 
understand  what  the  tool  is  and  is  not  designed  to  do.  The  purpose  of  the  tool  is  twofold:  to 
ensure  novice  site  selectors  are  cued  to  consider  the  full  range  of  appropriate  factors  for  site 
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selection,  and  to  discriminate  between  tasks  that  do  and  do  not  require  group  discussion.  The 
SST  is  not  expected  to  produce  an  “accurate”  site  recommendation  in  every  instance;  the 
presence  of  special  and  varied  circumstances  precludes  any  automated  system  from  producing  a 
correct  answer  every  time.  Instead,  CTSSBs  should  view  tasks  for  which  board  member 
opinions  do  not  coincide  with  SST  recommendations  as  ripe  for  discussion.  Rather  than  viewing 
these  instances  as  failures  by  the  SST,  board  members  should  consider  them  to  be  examples  of 
the  several  occasions  in  which  human  intelligence  trumps  the  workings  of  an  automated  system. 
Similarly,  in  instances  when  the  SST  reveals  a  lack  of  consensus  across  board  members,  the  task 
calls  for  group  discussion  to  understand  the  reasons  for  the  differences  of  opinion.  During  the 
assessment  phases,  board  members  reported  an  uncertainty  regarding  the  meaning  of 
flagged/highlighted  tasks.  Initially,  the  impression  was  that  the  SST  flagged  tasks  for  which 
board  members  had  answered  incorrectly  according  to  what  the  tool  ordained.  While  steps  have 
been  taken  to  resolve  this  impression  in  the  future  (e.g.,  tasks  are  no  longer  highlighted  red, 
which  is  often  associated  with  inaccuracy),  it  is  critical  that  the  CTSSB  chairperson  or 
coordinator  remind  board  members  of  the  SST’s  actual  purpose.  The  tool  makes 
recommendations  about  site  location  based  upon  board  member  inputs,  but  it  will  not  always 
provide  recommendations  with  which  the  board  members  agree.  It  is  expected  that  some  MOSs 
will  present  particularly  unique  training  environments,  and  for  these  MOSs,  a  higher  level  of 
disagreement  with  the  SST  may  exist  than  in  other  MOSs. 

In  applying  the  SST,  we  noted  broader  challenges  associated  with  the  conduct  of 
CTSSBs.  Leadership  of  the  board  is  critical.  With  or  without  automated  supports,  the  success  of 
board  outcomes  depends  on  effective  leadership  to  embrace  the  range  of  experiences  resident  in 
board  members  and  facilitate  a  healthy  debate  that  is  not  overtaken  by  a  few  strong  personalities. 
As  is  commonly  found  in  research  into  group  dynamics,  group  polarization  can  occur  such  that 
an  initial  small  majority  is  transfonned  into  a  substantial  majority  fairly  quickly  (Moscovici  & 
Zavalloni,  1969).  Dissenting  opinions  may  not  always  be  actively  sought,  and  if  individuals 
holding  dissenting  opinions  are  not  outspoken  or  assertive  individuals  in  this  setting,  the 
recorded  group  vote  may  represent  only  the  opinion  of  a  few  individuals  speaking  for  the  group. 
While  it  may  be  easier  for  a  single  board  member  to  dissent  against  the  group  when  he  or  she  is 
one  of  a  few  rather  than  one  of  many,  the  likelihood  of  having  another  dissenter  increases  with  a 
greater  number  of  people,  again  highlighting  the  complexity  of  greater  versus  fewer  board 
members.  In  this  same  vein,  research  has  repeatedly  demonstrated  that  voicing  one’s  opinion  in 
front  of  a  group,  when  others  have  already  made  their  opinions  known,  increases  the  likelihood 
that  an  individual  will  voice  the  existing  group  opinion  (i.e.,  conform  to  the  group’s  opinion; 
Asch,  1954).  We  observed  this  voting  process  several  times  during  this  research  effort.  To 
reduce  the  social  pressure  to  agree  with  other  board  members,  private  voting  is  a  more  reliable 
approach,  and  is  enabled  via  the  SST.  Of  course,  private  voting  can  be  more  time-consuming, 
particularly  when  boards  must  vote  on  several  hundred  tasks  (as  was  the  case  in  some  of  the 
observations  for  this  effort. 

The  quality  of  the  board  members  selected  to  represent  the  operating  force,  and  the 
knowledge  they  bring  to  the  board,  is  likewise  crucial  to  success.  The  SST  cannot  be  a  substitute 
for  lack  of  board  member  knowledge.  During  the  assessment  phase  of  this  research  effort,  some 
board  members  indicated  that  the  SST  was  not  able  to  account  for  the  complexity  of  operational 
experience,  and  that  the  board  members  could  only  vote  based  on  their  personal  experience, 
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which  would  likely  not  represent  the  full  array  of  potential  experiences  with  a  given  task.  While 
this  is  certainly  true,  this  criticism  holds  for  any  group  decision  making  process,  including  the 
criticality  voting  process.  The  usefulness  of  the  SST  is  only  as  good  as  the  expertise  of  the 
voting  members,  just  as  the  validity  of  the  criticality  vote  is  only  as  good  as  the  members’ 
expertise.  Some  test  users  indicated  a  preference  to  indicate  their  level  of  expertise  on  the  task, 
such  as  weighting  the  board  members’  votes  by  rank,  time  in  service,  and  so  forth.  Because  the 
board  composition  is  mandated  by  TRADOC  Pamphlet  350-70-6  and  the  same  issues  arise  in  the 
criticality  voting  phase,  the  decision  was  made  not  to  include  this  functionality  in  the  SST.  This 
inherent  limitation  in  human  decision  making  is  acknowledged,  and  to  counteract  it,  attempts  are 
made  to  recruit  board  members  from  a  vast  array  of  Anny  experiences.  For  example,  some  MOS 
CTSSBs  include  Soldiers  representing  both  commander’s  staff  positions  as  well  as  non-staff  duty 
positions  at  lower  echelons.  Individuals  whose  primary  experience  resides  at  a  staff  position 
would  likely  have  a  substantially  different  opinion  of  what  tasks  are  critical  and  where  they 
should  be  trained  than  individuals  who  experiences  are  in  other  duty  contexts.  Indeed,  SMEs 
during  our  research  acknowledged  that  variations  in  make  up  from  one  board  to  the  next  affects 
the  critical  task  list.  This  experience  presumably  also  affects  the  conceptualization  of  how  a  task 
is  performed  and  trained,  and  thus  opinions  about  training  site  location. 

One  board  we  observed  included  an  MOS  with  three  very  distinct  subgroups  of  expertise. 
For  the  criticality  vote,  this  board  divided  into  SME  subgroups  that  voted  on  the  criticality  of 
tasks  specific  to  their  subject  matter  expertise.  The  larger  board  then  deferred  to  the  respective 
subgroups  for  their  official  vote  of  a  task’s  criticality.  During  the  assessment  of  the  SST, 
however,  all  board  members  completed  ratings  of  all  tasks,  and  during  the  focus  group  they 
indicated  that  they  preferred  to  opt  out  of  rating  tasks  for  which  they  felt  they  had  no  expertise, 
despite  holding  the  relevant  MOS.  As  a  result  of  this  user  input,  the  SST  was  modified  to  allow 
for  users  to  opt  out  of  responding  to  particular  tasks  on  the  list.  Further,  the  implementation 
guidance  for  the  administrator  offers  advice  for  administrators  who  may  wish  to  fonn  subgroups 
of  board  members  with  varying  expertise  on  a  subset  of  critical  tasks.  In  brief,  the  administrator 
could  fonn  subgroups  and  populate  the  task  fields  of  these  subgroups  only  with  tasks  that 
matched  their  respective  expertise. 

As  mentioned  above,  some  users  expressed  a  desire  to  utilize  the  SST  prior  to  the  board 
convening,  much  as  AUTOGEN  is  intended  to  be  used  as  a  pre-assessment  of  task  criticality 
before  the  official  board  convenes.  Although  the  SST  was  not  conceptualized  to  be  applied  pre- 
CTSSB,  and  this  approach  faces  unique  obstacles  such  as  including  a  greater  number  of  tasks  to 
rate  (as  the  task  list  would  not  have  been  pared  down  to  only  the  critical  tasks),  potential  for 
technical  difficulties,  and  a  lack  of  an  administrator  to  guide  the  user  through  the  SST.  However, 
despite  these  challenges,  the  use  of  the  SST  pre-CSTTB  is  certainly  feasible.  Administrators 
may  wish  to  disseminate  the  tool  to  board  members  and  have  the  board  members  utilize  the  SST 
on  the  total  task  list  prior  to  arriving  for  the  board.  However,  as  CTSSBs  may  be  eventually 
conducted  solely  through  distributed  means  (via  telecon  or  web-based  collaboration  platforms), 
the  SST  would  easily  accommodate  such  a  transition. 

Although  the  Site  Selection  Tool  was  developed  with  a  focus  on  NCOES,  the  Army  uses 
the  same  CTSSB  process  for  OES  and  functional  courses,  as  well.  Generally,  the  tool  and  the 
results  of  this  analysis  are  expected  to  be  applicable  to  the  OES  as  well,  as  there  is  no  reason  to 
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believe  that  the  characteristics  that  distinguish  NCO  critical  tasks  best  suited  for  specific  training 
environments  differ  from  those  that  distinguish  Officer  critical  tasks.  Indeed,  the  same  process 
and  many  times  the  same  people  conduct  Officer  CTSSBs  as  do  NCO  CTSSBs,  and  this  research 
includes  feedback  from  individuals  familiar  with  both  Officer  and  NCO  CTSSBs.  Likewise,  the 
tool  may  also  be  utilized  for  functional  courses  (such  as  Ranger  School,  Bradley  Gunner  Course, 
or  Drill  Sergeant  School).  However,  the  purpose  of  a  CTSSB  for  a  functional  course  seems  to  be 
more  unique  in  that  these  boards  assess  which  tasks  should  be  included  in  the  functional  course 
POI  with  the  primary  decision  point  being  to  include  or  exclude  a  task  from  the  course’s  POI,  not 
a  decision  between  three  viable  options  (the  institutional  course,  the  unit,  or  self-development). 
As  such,  use  of  the  SST  in  functional  course  CTSSBs  should  be  considered  carefully,  although 
as  the  Army  Learning  Model  is  more  fully  implemented  and  more  interest  in  training  tasks  via 
self-development  or  distributed  learning  occurs  even  within  function  courses,  this  unique 
concern  may  diminish. 

Although  the  SST  and  the  CTSSB  process  generally  produce  recommendations  regarding 
the  training  site  of  critical  tasks,  these  recommendations  should  be  couched  within  the  broader 
concerns  of  an  Army  proponent’s  needs  and  resources.  Just  as  the  CTSSB  can  and  should 
overrule  the  SST  if  the  SST’s  recommendation  provides  a  faulty  mismatch  between  the  board’s 
understanding  of  a  task  and  the  SST’s  internal  logic,  the  final  CTSSB  recommendations  (after 
utilizing  the  SST)  can  and  should  be  considered  within  the  scope  of  the  Army’s  needs  as  training 
developers  and  commandants  understand  them.  A  CTSSB  provides  training  site 
recommendations  based  upon  its  understanding  of  the  tasks,  utilization  of  the  SST,  and 
discussion.  These  determinations  represent  the  best  recommendation  as  a  function  of  the  board 
members’  knowledge  of  task,  operational  experience,  training  experiences,  etc.,  and  are  based 
upon  an  ideal  match  between  the  task  and  training  environment.  However,  an  ideal  match  must 
be  filtered  through  real  constraints  so  the  prepared  list  of  training  site  recommendations  is  then 
assessed  by  the  course  developers.  These  course  developers  then  apply  their  knowledge  of 
resource  constraints  and  integration  ability  with  others  tasks  in  the  POI  to  make  a  final 
recommendation  to  the  commandant.  The  commandant  then  considers  additional  factors  such  as 
practical  and  resource  considerations  and  higher-level  Army  guidance  or  other  information  to 
which  board  members  and  course  developers  are  not  privy.  Although  the  commandant  may 
generally  rely  upon  the  operational  and  front-line  expertise  of  board  members  who  have  recently 
performed  or  supervised  perfonnance  of  the  critical  task,  there  may  not  always  be  a  match 
between  the  board’s  recommendations  and  the  proponent’s  understanding  of  the  realities  of 
training.  The  commandant’s  approval  triggers  any  relevant  changes  to  course  curricula  and 
POIs. 

Limitations  of  the  SST 

Although  every  effort  was  made  to  maximize  the  quality,  usefulness,  and  user- 
friendliness  of  the  SST,  some  limitations  remain,  as  is  true  for  any  tool.  Some  limitations  are  a 
function  of  human  behavior  and  some  are  inherent  to  the  technical  constraints. 

Gaming  the  system.  Several  measures  were  taken  to  decrease  the  likelihood  that  users 
would  game  the  system  in  attempts  to  have  the  SST  output  a  recommendation  to  match  his  or  her 
personal  opinion.  To  that  end,  the  research  team  implemented  new  functionality  to  enable  users 
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to  disagree  with  the  system  and  identify  their  personal  recommendation.  Nonetheless,  users  still 
attempted  to  game  the  system.  It  is  not  clear  whether  this  has  an  impact  on  the  final  site 
selection  recommendations  submitted  by  the  CTSSBs.  Presumably,  this  effect  would  be 
mitigated  by  the  group  discussion  that  occurs  when  board  members  fail  to  reach  consensus  on  a 
task.  An  attempt  has  been  made  to  reduce  this  tendency  to  game  the  system  by  including  a  note 
to  the  users  in  the  administrator’s  guide.  This  note  reminds  users  that  gaming  the  system 
effectively  introduces  random  error  noise  into  the  tool’s  process  and  likely  reduces  the  ability  of 
the  group,  via  the  SST,  to  reach  a  consensus.  Board  chairs  and  administrators  should  likewise 
emphasize  this  caution.  An  advisory  warning  explains  this  concern  in  the  implementation  guide 
(Appendix  H)  in  layman’s  terms. 

Consensus  level.  CTSSBs  vary  in  the  number  of  voting  members,  with  most  boards 
consisting  of  six  to  ten  individuals.  With  larger  boards,  a  single  dissenting  vote  represents  a 
smaller  percentage  of  the  potential  group  consensus  (e.g.,  10%  for  groups  of  ten,  and  17%  for 
groups  of  six).  Such  variations  in  the  impact  of  a  single  (or  a  couple  of)  dissenting  votes  means 
that  at  times,  reaching  the  requisite  level  of  consensus  can  be  more  difficult  for  smaller  boards 
than  for  larger  boards,  so  board  administrators  may  wish  to  set  a  lower  consensus  level  that  is 
more  easily  attained  with  a  smaller  board.  Likewise,  variations  exist  across  MOSs  and  by  board 
chairpersons  regarding  what  level  of  consensus  is  sufficient  and  appropriate.  Some  boards  may 
prefer  a  simple  majority,  while  others  may  prefer  a  higher  level  of  agreement.  During  the 
development  and  assessment  of  the  tool,  we  utilized  a  65%  level  of  agreement,  meaning  that  if 
65%  of  users  agree,  then  the  task  was  not  flagged  for  discussion.  Based  on  feedback  from  users 
during  the  assessment  phase,  we  modified  the  SST  to  include  the  ability  to  select  a  consensus 
level  that  best  matched  the  board’s  preferences.  In  the  current  iteration,  the  SST  offers  a 
selection  of  options  for  consensus  level — 50%,  60%,  70%,  or  80%. 

Technical  Features.  Every  effort  was  made  to  ease  the  workload  of  the  SST 
administrator  in  compiling  data  across  board  members.  However,  we  ultimately  sacrificed 
technical  functionality  for  the  ease  of  access  and  immediacy  of  use  conferred  by  Microsoft 
Excel’s  standing  Certificate  of  Networthiness  (CON)  with  the  Anny’s  Office  of  Information 
Assurance  and  Compliance.  Security  requirements  and  limitations  within  the  Microsoft  Excel 
platform  prevent  the  ability  to  automatically  compile  board  member  data  without  a  basic  level  of 
manipulation  by  the  administrator  to  enable  macros  within  the  SST  file.  Ideally,  security 
restrictions  would  not  require  behind-the-scenes  macro-enabling  in  order  to  compile  data. 
However,  this  is  unavoidable  given  the  current  restrictions  in  place  at  most  Anny  installations. 
As  the  Army  updates  its  computer  security  policies  and  as  the  Microsoft  Excel  platfonn  is 
updated,  the  current  iteration  of  the  SST  may  require  updates  to  keep  pace  with  the  other 
advancements  and  developing  restrictions.  Although  this  dependency  on  the  CON  is  limiting  in 
some  regards,  it  is  still  seen  as  advantageous  over  obtaining  a  separate  CON  for  the  SST  that 
requires  separate,  independent  oversight. 

Task  Endurability 

Originally,  the  research  team  expected  that  task  endurability  would  be  a  prevailing 
distinguisher  between  tasks  suited  for  institutional,  unit,  or  self-development  training.  However, 
the  concept  of  task  endurability  proved  difficult  to  address,  with  SMEs  exhibiting  a  great  deal  of 
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variability  in  their  ideas  about  enduring  and  non-enduring  tasks,  as  well  as  their  specification  as 
to  what  makes  a  task  enduring  versus  non-enduring.  Enduring  tasks  were  most  often  associated 
with  those  that  were  doctrinal  or  general  in  nature,  while  non-enduring  tasks  were  often 
conceptualized  as  those  that  were  TTP-driven  or  theater  specific,  or  subject  to  changes  in 
technologies  or  equipment.  In  outlining  and  describing  the  characteristics  of  a  task  that  make  it 
best  suited  for  an  institutional  environment  versus  a  unit  or  self-development  environment, 
endurability  (or  close  proxies)  was  generally  not  reported  as  a  distinguishing  characteristic. 

When  pressed,  SMEs  did  acknowledge  that  endurability  was  theoretically  an  appropriate 
distinction,  and  could  speculate  as  to  tasks  best  suited  to  different  training  environments  on  the 
basis  of  endurability.  However,  the  data  analysis  revealed  endurability  to  be  more  appropriately 
defined  as  a  family  of  related  factors  vice  a  single  factor. 

Generally,  the  constellation  of  factors  associated  with  task  endurability  was  not  viewed 
by  SMEs  to  necessarily  dictate  site  placement.  In  other  words,  arguments  can  be  reasonably 
made  for  training  a  particular  non-enduring  task  in  the  institution.  One  of  the  perceived  benefits 
of  institutional  training  is  its  ability  to  quickly  reach  a  broad,  captive  audience  and  provide 
standardized  instruction  across  the  population.  Although  the  typical  course  development  cycle 
updates  approximately  every  three  years,  the  Anny  can  rapidly  produce  mandatory  training 
modules  to  be  included  in  courses  outside  of  the  official  POI.  It  follows  that  crucial  threats  to 
human  life,  such  as  improvised  explosive  devices  (IEDs)  or  Soldier  suicides,  might  be 
appropriate  topics  for  institutional  training.  In  these  cases,  human  judgment  as  to  the  degree  of 
the  task’s  endurability  comes  into  the  equation.  One  might  argue  that  a  task  that  endures  for  ten 
years  vice  two  years  is  appropriate  for  institutional  training.  The  onus  then  is  on  the  institution 
to  note  when  this  training  is  no  longer  necessary  and  should  be  removed  from  course  POIs. 

However,  many  SMEs  reported  their  frustration  with  training  modules  that  are  mandated 
for  inclusion  in  their  course,  when  identical  training  is  also  mandated  at  the  unit  and  in  other 
courses.  The  NCOs  receive  the  same  training  content  across  multiple  locations  and  stages  of 
their  career,  with  no  adjustment  for  skill  level.  The  SMEs  perceive  that  after  the  first  iteration  of 
the  training,  there  is  little  to  no  value  in  presenting  the  training  again  and  again,  and  it  comes  at 
the  expense  of  training  other  important  tasks.  Two  prime  examples  are  suicide  prevention  and 
sexual  harassment  training.  In  these  instances,  the  frustration  is  not  with  the  judgment  that  the 
topics  are  critically  important.  The  SMEs  perceive  that  training  time  could  be  better  used  by 
either  introducing  advanced  concepts  within  the  topic  to  maintain  trainee  attention  and 
engagement,  or  foregoing  the  training  entirely  in  service  of  training  that  addresses  new 
knowledge  or  skills.  Mandates  for  particular  Anny-wide  programs  of  instruction  reveal  a 
perception  of  the  institution  as  a  primary  and  valuable  vehicle  for  delivering  required  training  to 
the  Anny  at  large  in  a  quick  and  controlled  manner,  regardless  of  its  relevance  to  the  course 
subject  matter. 

One  strategy  for  addressing  the  need  to  accommodate  tasks  with  changing  parameters  and 
conditions  by  one  of  the  NCO  Academies  we  observed  was  an  effort  to  re-write  the  MOS’s  tasks 
to  weather  the  equipment  and  technology  changes  over  the  course  of  time.  The  wording  of  the 
task  would  focus  on  the  desired  outcome  of  the  task  vice  utilization  of  a  particular  system  in 
service  of  that  outcome.  In  this  way,  the  tasks’  underlying  purpose  and  general  skill  would  not 
need  to  change  but  the  specifics  for  institutional  or  unit  instruction  has  leeway  to  adapt 
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appropriately.  To  use  a  fictitious  example,  the  task  “Employ  the  ABC  system  to  identify  trends” 
would  be  rewritten  as  “Conduct  trend  analysis  to  identify  trends.”  Such  rewording  also 
accommodates  for  differences  in  the  equipment  owned  and  used  by  units  across  the  operating 
forces.  The  downside  of  this  approach  is  that,  while  building  in  flexibility  to  allow  for  access  to 
varied  types  of  equipment,  units  lose  the  ability  to  safely  assume  that  newly  assigned  NCOs  have 
been  trained  on  the  same  equipment. 

Of  particular  interest  in  relation  to  endurability  is  the  concept  of  emerging  tasks,  vice 
enduring  tasks.  Emerging  tasks  might  be  defined  as  those  in  support  of  new  mission 
requirements  that  reflect  evolutions  to  the  ways  in  which  wars  are  fought  or  other  military 
operations  are  conducted.  For  example,  sensitive  site  exploitation  and  forensic  analysis  are  now 
being  perfonned  by  infantry  and  other  units  in  response  to  the  tactics  and  practices  of  most  all 
adversary  networks  encountered  by  Army  forces,  and  given  the  ability  of  technological  advances 
to  support  such  activities.  Similarly,  tasks  once  reserved  for  a  particular  MOS  have  migrated  to 
be  more  broadly  applied  by  a  range  of  MOSs.  The  biometrics  collections  once  specific  to 
Military  Police  MOSs  are  now  conducted  by  infantry  and  other  MOSs.  Intelligence  tasks 
including  the  collection  of  human  intelligence  through  tactical  questioning  and  engagements 
with  local  leaders  is  likewise  conducted  by  individuals  representing  a  broader  range  of  MOSs. 
These  types  of  tasks  have  emerged  and  will  arguably  be  sustained  for  the  foreseeable  future, 
across  operational  and  mission  environments.  They  do  not  fit  the  mold  of  non-enduring  tasks 
that  are  specific  to  a  particular  region  or  adversary.  Since  global  dynamics  are  continuously 
changing,  and  along  with  them  the  role  of  the  U.S.  military,  the  concept  of  tasks  that  emerge  and 
evolve  may  be  more  fitting  than  tasks  that  are  either  pennanent  or  temporary.  As  such,  the 
traditional  distinction  and  appropriation  of  tasks  into  instruction  versus  unit  versus  self-study 
may  evolve.  If  the  military  training  landscape  begins  to  face  more  turbulent  turnover  of  critical 
tasks,  the  role  of  unit  training  will  face  an  increasing  burden  as  the  trainer  of  new  tasks  and  the 
rapid  responder  to  the  changing  operational  environment. 

Future  Development 

Although  we  conducted  several  assessments  and  attempted  to  gamer  representation 
across  a  wide  range  of  Army  MOSs  and  experiences,  we  were  unable  to  assess  the  SST  under 
every  possible  MOS.  The  current  version  of  the  SST  represents  the  combined  lessons  learned 
from  the  groups  that  were  assessed,  representing  subject  matter  expertise  with  many  years  of 
experience  and  varied  Army  backgrounds.  However,  future  iterations  may  include  additional 
features  that  come  to  light  as  necessary.  Through  the  course  of  this  project,  a  few  proposed 
changes  and  feature  requests  were  not  included  as  they  were  either  considered  to  be  outside  the 
scope  of  the  project,  or  consensus  did  not  exist  regarding  the  need  for  the  modification  to  suit 
many  MOSs. 

One  such  modification  is  the  ability  to  customize  the  institutional  site  recommendation  to 
display  the  specific  course  under  review,  such  as  Warrior  Leaders  Course  or  Infantry  ALC,  as 
opposed  to  the  generic  “institution.”  In  observations  of  boards  both  before  and  after  the 
implementation  of  the  SST,  the  research  team  observed  occasional  line-blurring  between 
NCOES/OES  courses  and  the  functional  courses  offered  by  the  institution.  For  example,  the 
Squad  Designated  Marksmanship  Course  is  not  within  the  scope  of  NCOES,  but  it  is  an 
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institutional  course  in  the  sense  that  is  offered  by  Anny  Marksmanship  Unit  at  Fort  Benning, 

GA,  and  it  is  similar  to  NCOES  structure  in  that  it  has  a  fairly  fonnal  structure.  With  the 
addition  of  a  feature  that  specified  and  reminded  SST  users  that  the  tool  identified  “institutional” 
as  Annor  ALC  rather  than  Master  Gunner  Course,  there  would  be  less  potential  for  confusion 
when  discussing  the  ideal  training  site  for  related  critical  tasks.  Currently,  the  board 
administrator  has  the  ability  to  type  the  name  of  the  course  under  evaluation  into  a  field  at  the  top 
of  the  SST.  While  this  will  help  reduce  confusion  about  which  course  is  meant  by  “institution,” 
a  reminder  after  every  task’s  site  determination  recommendation  would  be  a  clearer  and  more 
frequent  reminder  for  the  users. 

For  future  iterations,  it  will  also  be  fruitful  to  carefully  consider  low-density  MOSs.  Low 
density  MOSs  in  highly  specific  technical  skills  may  find  themselves  dispersed  broadly,  serving 
as  a  single  individual  attached  to  a  broader  unit.  For  example,  an  individual  with  a  military 
intelligence  MOS  may  be  attached  to  an  infantry  company.  In  such  a  situation,  his  “unit”  may  be 
dispersed  over  many  other  companies,  and  he  has  limited  contact  with  his  parent  unit.  Therefore, 
“unit  training”  is  not  regularly  available  in  any  meaningful  sense;  SMEs  may  be  assigned  to  the 
unit,  but  are  not  accessible  within  reasonable  time  and  resources.  In  the  current  iteration,  we 
believe  that  the  current  wording  of  the  questions  still  adequately  addresses  the  availability  of  unit 
resources  and  SMEs.  However,  with  additional  feedback  and  exposure  to  other  MOSs, 
necessary  adaptations  may  come  to  light. 

Another  potential  point  of  departure  for  future  SST  iterations  will  be  the  manner  in  which 
self-development  is  encouraged  and/or  delivered  in  the  Anny.  Strictly  speaking,  self¬ 
development  includes  any  training  that  is  not  conducted  by  a  fonnal  instructor  at  the  institution 
or  delivered  by  a  face-to-face  interaction  with  a  unit  trainer.  Cunently,  self-development  is 
construed  to  mean  Anny-developed  dL.  However,  this  is  not  necessarily  the  case,  as  self¬ 
development  should  also  include  non-structured  self-learning  by  motivated  individuals  who  seek 
out  expert  sources  (e.g.,  individuals  or  formal  training  manuals  and  documents)  without  the 
guidance  of  an  online  course.  The  current  iteration  of  the  SST  relied  on  the  current 
conceptualization  of  self-development  as  fonnal  web-based  training.  As  the  Anny  continues  to 
evolve  its  interpretation  and  expectations  for  self-development,  the  logic  guiding  the  SST’s 
placement  of  tasks  into  self-development  will  need  to  be  revisited.  Technology  advances  and 
accessibility  will  also  influence  the  amount  and  type  of  self-development  possible. 

As  technology  continues  to  play  an  increasingly  important  role  in  Army  training,  a 
greater  number  of  tasks  can  be  expected  to  be  trained  using  simulations.  Depending  upon  the 
complexity  and  expense  of  the  technology  required  to  conduct  simulated  training,  the  ubiquity  of 
access  to  the  necessary  training  equipment  will  vary.  The  underlying  logic  and  process  for 
assigning  tasks  to  self-development,  unit,  or  institutional  training  environments  partially  relies 
upon  accessibility  of  the  necessary  equipment.  The  current  version  of  the  SST  distinguishes 
between  actual  equipment  and  an  “equipment  substitute;”  if  access  to  the  actual  equipment  is 
required,  then  the  task  should  not  be  trained  in  self-development.  However,  as  simulation 
technology  continues  to  develop  and  offer  higher  fidelity  simulations  at  lower  cost,  likely 
increasing  availability  of  and  access  to  simulation  technology,  this  factor’s  value  to  site 
determination  may  be  impacted.  Likewise,  some  MOSs,  such  as  those  that  rely  heavily  on 
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computers,  may  place  less  value  in  distinguishing  between  actual  technology  and  simulated 
technology  in  that  there  may  be  little  difference  from  the  perspective  of  the  Soldier  training. 

During  the  criticality  vote  of  the  boards  we  observed,  board  members  frequently  raised 
questions  about  the  content,  conditions,  and  standards  of  particular  tasks.  Administrators  and 
their  personnel  generally  had  reference  resources  available  to  answer  these  questions  from 
official  Anny  documentation.  Along  these  same  lines,  some  users  indicated  that  they  would  find 
a  similar  capability  within  the  SST  to  be  helpful,  such  as  hyperlinks  from  the  tool  to  a  website 
with  appropriate  task  definitions  and  descriptions.  Future  iterations  of  this  tool  may  explore  the 
inclusion  of  this  functionality.  However,  in  its  current  version,  the  SST  could  easily 
accommodate  administrators’  manual  creation  of  hyperlinks  from  task  titles  listed  in  the  task 
fields  to  the  task  reference,  using  Microsoft  Excel  hyperlink  functionality.  Moreover,  the  Anny 
may  wish  to  develop  a  suite  of  web-accessible  CTSSB  tools,  to  include  not  only  the  SST  and 
links  to  task  definitions  and  descriptions,  but  also  the  task  selection  models  and  the  CTSSB  tools 
used  to  vote  on  critical  tasks.  Additionally,  access  to  CTSSB  results  are  of  interest  to  many 
training  community  members.  As  the  SST  outputs  are  Excel  format  spreadsheets,  these  results 
can  be  easily  shared  not  only  informally  via  email,  CDs,  etc.,  but  may  also  be  more  widely  and 
easily  available  by  storing  the  outcomes  in  the  newly  developed  Training  Development 
Capability  (TDC).  TDC  is  an  Army-wide  database  for  storing  training  information  associated 
with  various  tasks  such  as  training  manuals,  POIs,  tutorials,  trainer  resources,  etc.  Each  task  in 
TDC  is  maintained  by  the  appropriate  task  proponent  to  ensure  currency  of  content,  and  the  SST 
outcomes  could  be  easily  added  to  the  repository  of  infonnation  available  in  TDC. 

Toward  the  goal  of  improving  site  selection  NCOES,  the  SST  will  be  made  available  as 
widely  as  possible.  The  sponsor  for  this  research  effort,  the  INCOPD,  will  retain  copies  of  the 
SST  files  and  user  guides.  Although  the  tool  was  developed  primarily  for  use  in  NCOES 
training  decision-making,  the  tool  can  also  be  used  for  OES  and  functional  course  training 
decisions.  The  tool  will  also  be  available  for  download  on  the  Army  Training  Network  and 
through  the  U.S.  Army  Research  Institute.  Included  in  the  Appendices  are  user  guides  to  support 
technical  implementation  and  tips  and  suggestions  for  incorporating  the  SST  into  CTSSB 
process. 
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Acronyms 


ABOLC 

ADDIE 

AIT 

ALC 

ALM 

AKO 

ARI 

CIED 

CON 

CPT 

CTSSB 

dL 

DoD 

FEA 

IED 

INCOPD 

ITE 

KSAs 

LT 

LTC 

MAJ 

MOS 

MSG 

NCO 

NCOA 

NCOES 

OES 

POI 

SFC 

SLC 

SMC 


Armor  Basic  Officer  Leadership  Course 

Analysis,  Design,  Development,  Implementation,  and  Evaluation 

Advanced  Individual  Training 

Advanced  Leaders  Course 

Anny  Learning  Model 

Anny  Knowledge  Online 

Anny  Research  Institute 

Counter  Improvised  Explosive  Device 
Certificate  of  Networthiness 
Captain 

Critical  Task  and  Site  Selection  Board 

Distributed  Learning 
Department  of  Defense 

Front  End  Analysis 

Improvised  Explosive  Device 

Institute  for  Noncommissioned  Officers  Professional  Development 
Integrated  Training  Environment 

Knowledge,  Skills,  and  Abilities 

Lieutenant 
Lieutenant  Colonel 

Major 

Military  Occupational  Specialty 
Master  Sergeant 

Noncommissioned  Officer 
Noncommissioned  Officer  Academy 
Noncommissioned  Officer  Educational  System 

Officer  Education  System 

Program  of  Instruction 

Sergeant  First  Class 
Senior  Leaders  Course 
Sergeant  Majors  Course 
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SME 

Subject  Matter  Expert 

SOPs 

Standard  Operating  Procedures 

SSG 

Staff  Sergeant 

SST 

Site  Selection  Tool 

TDC 

Training  Development  Capability 

TRADOC 

Training  and  Doctrine  Command 

TTPs 

Tactics,  Techniques  and  Procedures 

USASMA 

United  States  Army  Sergeant  Majors  Academy 

WLC 

Warrior  Leaders  Course 
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Appendix  A.  Informed  Consent  Form 


INFORMED  CONSENT 
(SME  Interviews) 

Title:  Front-End  Analysis  Methods  for  Noncommissioned  Officer  Education  System  (NCOES) 

Purpose  of  the  research  study:  The  objective  of  this  effort  is  to  identify  the  optimal 
placement  of  task  training.  The  research  team  will  be  developing  a  front-end  analysis 
methodology  to  differentiate  enduring  tasks  needed  to  develop  leaders  and  Soldiers  at  the 
different  levels  of  the  NCOES  from  non-enduring  tasks  needed  to  conduct  current  and  upcoming 
operations. 

What  you  will  be  asked  to  do  in  this  study:  An  interviewer  will  ask  you  a  series  of 
questions,  based  on  your  personal  experiences,  about  the  factors  and  characteristics  you  consider 
when  distinguishing  between  enduring  and  non-enduring  tasks.  Your  responses  will  be  recorded 
via  a  digital  audio  recording  device,  if  you  consent  to  it.  The  purpose  of  the  recording  is  to 
ensure  we  do  not  miss  or  misrepresent  any  of  the  infonnation  you  provide.  We  will  use  the  audio 
recordings  only  to  verify  what  we  have  written  in  our  interview  notes.  Your  personal  identifying 
information  will  not  be  maintained  with  the  recording,  and  the  recording  will  not  be  available  to 
anyone  outside  the  project  team. 

Location:  This  study  will  be  conducted  at  Fort  Benning,  GA;  Fort  Bliss,  TX;  and  Fort 
Huachuca,  AZ.  Telephonic  interviews  with  individuals  within  the  contiguous  United  States  will 
also  contribute  to  the  study. 

Time  required:  May  last  up  to  60-90  minutes,  but  no  longer  than  that. 

Classified  Information:  This  interview  will  be  conducted  at  an  UNCLASSIFIED  level. 
For  Official  Use  Only  (FOUO)  infonnation  may  be  discussed.  If  during  the  course  of  the 
interview  or  after  the  conclusion  of  the  interview,  you  feel  you  have  revealed  classified 
information,  please  infonn  the  interviewer  so  that  any  capture  of  that  information  can  be 
destroyed  immediately. 

Anonymity:  All  personal  infonnation  gathered  during  this  study  will  be  kept  strictly 
confidential  in  accordance  with  the  DoD  Privacy  Program  described  in  DoD  Directive  5400.11. 
Your  name  will  not  be  directly  associated  with  any  data.  Further,  the  information  provided 
throughout  participation  in  this  study  will  be  stored  in  such  a  way  that  the  data  cannot  be 
connected  to  any  individual,  thus  ensuring  privacy.  The  published  results  will  not  include  your 
name  or  any  other  infonnation  that  would  personally  identify  you  in  any  way. 

Voluntary  participation:  The  interviewers  agree  to  answer  any  questions  that  you  may 
have  at  this  time  or  at  any  time  during  the  duration  of  the  study.  You  do  not  have  to  answer  any 
question  that  you  do  not  wish  to  answer.  Participation  is  voluntary  and  you  may  terminate 
participation  in  the  study  at  any  time  with  no  penalties.  If  at  anytime  during  the  study  you  feel 
uncomfortable  in  any  way,  you  can  and  should  inform  the  interviewer  and  the  study  will  be 
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terminated  immediately.  If  we  feel  that  participation  is  emotionally  stressful  for  you,  we  will  ask 
you  if  you  wish  to  stop  the  interview. 

Risks.  There  are  no  foreseeable  risks  for  participating  in  this  research. 

Benefits.  There  are  no  direct  benefits  to  you  for  participating  in  this  research.  Indirectly, 
you  will  be  contributing  to  an  improved  Army  process  for  better  distinguishing  between 
enduring  and  non-enduring  tasks.  This  will  aid  Anny  training  decisions. 

Compensation.  There  will  be  no  compensation  for  your  participation. 

Questions  regarding  your  rights  as  a  research  volunteer  may  be  sent  to 
AR1  RES@conus.army.mil .  (type:  “Front-End  Analysis  Methods  for  NCOES,  2011”  in  the 
subject  line). 

Questions  about  anything  having  to  do  with  this  study  can  be  sent  to 

ARI  RES@conus.army.mil .  (type:  “Front-End  Analysis  Methods  for  NCOES,  201 1”  in  the 
subject  line). 


If  responding  to  any  of  the  questions  becomes  unpleasant  for  you,  you  can  withdraw  from  the 
conversation  at  any  time.  Please  note  the  numbers  on  the  card  we  handed  out.  If  you  feel 
you’d  like  to  confer  with  someone  confidentially  after  this  discussion,  please  go  to  the 
Military  OneSource  web  site  (https://www.militarvonesource.com)  or  call  the  1-800-342- 
9647  number. 
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Title:  Front-End  Analysis  Methods  for  Noncommissioned  Officer  Education  System  (NCOES) 


If  you  agree  to  participate  in  this  study,  please  sign  and  date  below. 
Agreement:  I  have  read  the  procedures  described  above. 

_ I  am  at  least  1 8  years  of  age  (check) 

_ I  voluntarily  agree  to  participate  in  the  study  (check) 

_ I  consent  to  an  audio  recording  of  my  interview  (check) 

Signature:  _ Date: _ 
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INFORMED  CONSENT 
(Site  Selection  Tool  Assessment) 


Title:  Front-End  Analysis  Methods  for  Noncommissioned  Officer  Education  System  (NCOES) 

Purpose  of  the  research  study:  The  objective  of  this  effort  is  to  identify  the  optimal 
placement  of  task  training.  The  research  team  will  be  developing  a  front-end  analysis 
methodology  that  will  augment  the  current  task  analysis  and  site  selection  procedures,  and 
support  the  development  of  recommendations  about  enduring  tasks  to  be  considered  for 
institutional  training  vice  non-enduring  tasks  that  should  be  trained  at  home  station 

What  you  will  be  asked  to  do  in  this  study:  During  the  Critical  Task  and  Site  Selection 
Board  (CTSSB),  a  FEA  methodology  in  the  fonn  of  a  Site  Selection  Tool  will  be  implemented  to 
assist  you  with  site  selection  as  part  of  the  critical  task  review.  Once  the  analysis  is  completed, 
you  will  be  asked  to  complete  a  survey  to  provide  your  feedback  on  the  Site  Selection  Tool. 
Following  your  voluntary  completion  of  the  survey,  an  interviewer  will  ask  you  a  series  of 
questions  concerning  the  tools  functionality.  Your  responses  may  be  recorded  via  a  digital  audio 
recording  device,  if  you  consent  to  it.  The  purpose  of  the  recording  is  to  ensure  we  do  not  miss 
or  misrepresent  any  of  the  infonnation  you  provide.  We  will  use  the  audio  recordings  only  to 
verify  what  we  have  written  in  our  interview  notes.  Your  personal  identifying  infonnation  will 
not  be  maintained  with  the  recording,  and  the  recording  will  not  be  available  to  anyone  outside 
the  project  team. 

Location:  This  study  will  be  conducted  at  <insert  site  here>.  Telephonic  interviews  with 
individuals  within  the  contiguous  United  States  will  also  contribute  to  the  study. 

Time  required:  May  last  up  to  45  minutes,  but  no  longer  than  that. 

Classified  Information:  This  interview  will  be  conducted  at  an  UNCLASSIFIED  level. 
For  Official  Use  Only  (FOUO)  information  may  be  discussed.  If  during  the  course  of  the 
interview  or  after  the  conclusion  of  the  interview,  you  feel  you  have  revealed  classified 
infonnation,  please  infonn  the  interviewer  so  that  any  capture  of  that  information  can  be 
destroyed  immediately. 

Anonymity:  All  personal  infonnation  gathered  during  this  study  will  be  kept  strictly 
confidential  in  accordance  with  the  DoD  Privacy  Program  described  in  DoD  Directive  5400.11. 
Your  name  will  not  be  directly  associated  with  any  data.  Further,  the  infonnation  provided 
throughout  participation  in  this  study  will  be  stored  in  such  a  way  that  the  data  cannot  be 
connected  to  any  individual,  thus  ensuring  privacy.  The  published  results  will  not  include  your 
name  or  any  other  infonnation  that  would  personally  identify  you  in  any  way. 

Voluntary  participation:  The  interviewers  agree  to  answer  any  questions  that  you  may 
have  at  this  time  or  at  any  time  during  the  duration  of  the  study.  You  do  not  have  to  answer  any 
question  that  you  do  not  wish  to  answer.  Participation  is  voluntary  and  you  may  tenninate 
participation  in  the  study  at  any  time  with  no  penalties.  If  at  anytime  during  the  study  you  feel 
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uncomfortable  in  any  way,  you  can  and  should  inform  the  interviewer  and  the  study  will  be 
terminated  immediately.  If  we  feel  that  participation  is  emotionally  stressful  for  you,  we  will  ask 
you  if  you  wish  to  stop  the  interview. 

Risks.  There  are  no  foreseeable  risks  for  participating  in  this  research. 

Benefits.  There  are  no  direct  benefits  to  you  for  participating  in  this  research.  Indirectly, 
you  will  be  contributing  to  an  improved  Army  process  for  better  distinguishing  between 
enduring  and  non-enduring  tasks.  This  will  aid  Army  training  decisions. 

Compensation.  There  will  be  no  compensation  for  your  participation. 

Questions  regarding  your  rights  as  a  research  volunteer  may  be  sent  to 
ARI  RES@conus.army.mil .  (type:  “Front-End  Analysis  Methods  for  NCOES,  2011”  in  the 
subject  line). 

Questions  about  anything  having  to  do  with  this  study  can  be  sent  to 
ARI  RES@conus.army.mil .  (type:  “Front-End  Analysis  Methods  for  NCOES,  2011”  in  the 
subject  line). 


If  responding  to  any  of  the  questions  becomes  unpleasant  for  you,  you  can  withdraw  from  the 
conversation  at  any  time.  Please  note  the  numbers  on  the  card  we  handed  out.  If  you  feel 
you’d  like  to  confer  with  someone  confidentially  after  this  discussion,  please  go  to  the 
Military  OneSource  web  site  (https://www.militaryonesource.com)  or  call  the  1-800-342- 
9647  number. 
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Title:  Front-End  Analysis  Methods  for  Noncommissioned  Officer  Education  System  (NCOES) 


If  you  agree  to  participate  in  this  study,  please  sign  and  date  below. 
Agreement:  I  have  read  the  procedures  described  above. 

_ I  am  at  least  1 8  years  of  age  (check) 

_ I  voluntarily  agree  to  participate  in  the  study  (check) 

_ I  consent  to  an  audio  recording  of  my  interview  (check) 

Signature:  _ Date: _ 
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Appendix  B.  Privacy  Act  Statement 

PRIVACY  ACT  STATEMENT 

Project  Title:  Front-End  Analysis  Methods  for  Noncommissioned  Officer  Education  System 
(NCOES) 

Authority:  The  Department  of  the  Army  may  collect  the  information  requested  in  this 
session  under  the  authority  of  10  United  States  Code,  Section  2358,  “Research  and  Development 
Projects.”  In  accordance  with  the  Privacy  Act  of  1974  (Public  Law  93-579),  this  notice  informs 
you  of  the  purpose,  use,  and  confidentiality  of  this  session. 

Purpose:  This  research  will  develop  a  front-end  analysis  methodology  for  identifying 
those  enduring  tasks  and  procedures  suitable  for  NCOES  instruction  and  those  changing  tasks 
and  procedures  suitable  for  unit  and  pre-deployment  training. 

Routine  Uses:  Information  from  this  project  will  be  used  to  infonn  a  design  of  future 
research  materials  and  potential  training  methods/products/tools  will  be  developed  to  determine 
which  types  of  tasks  are  less  enduring  for  multiple  mission  sets,  and  thus  better  suited  to  be 
trained  at  the  unit  level  or  via  individual  self-development.  Improvements  may  result  directly 
from  experiences  and  responses  you  and  your  fellow  Soldiers  provide. 

Disclosure:  Participating  in  this  project  is  voluntary  and  you  may  choose  at  any  time  not 
to  participate.  There  is  no  penalty  for  choosing  not  to  participate  and  you  may  leave  at  any  time. 
You  also  have  the  right  to  have  any  information  we  collect  from  you  to  be  withheld  from  further 
analysis. 

Confidentiality:  We  will  not  identify  you,  or  attribute  comments  made  during  the  project 
to  any  particular  participant,  and  we  will  not  include  your  name  or  other  personally  identifiable 
information  in  our  interview  notes  or  potential  reports.  Likewise,  we  ask  that  each  of  you  respect 
the  confidential  nature  of  this  project,  by  not  identifying  individual  participants  with  comments 
made  or  heard  during  this  session. 

We  cannot  provide  "confidentiality"  or  "non- attribution,"  to  a  participant  regarding  comments 
involving  criminal  activity/behavior,  or  statements  that  pose  a  threat  to  yourself  or  others.  Do  not 
discuss  or  comment  on  classified  or  operationally  sensitive  information  during  this  session. 

Contact:  For  further  information  about  this  project  or  your  rights  as  a  participant,  send  e- 
mail  to:  ARI  RES@conus.army.mil  (type  “Front-End  Analysis  Methods  for  NCOES,  2011”  in 
the  subject  line). 


RETAIN  FOR  YOUR  RECORDS 
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Appendix  C.  Data  Collection  Protocol 


INTERVIEW  PROTOCOL 

Front-End  Analysis  for  the  NCOES 
SME  Interview  Protocol 


Objectives 

1 .  Identify  all  factors,  including  those  related  to  task  endurability,  currently  used  to 
detennine  site  placement. 

2.  Identify  the  doctrinal  processes  and  experiential  knowledge  currently  used  to  think  of  a 
task  as  enduring  or  non-enduring. 

3.  Identify  additional  variables,  factors,  and/or  characteristics  of  enduring  versus  non¬ 
enduring  tasks  that  could  be  considered  to  improve  the  efficiency  of  training. 

4.  Identify  one  or  more  entry  point  opportunity  for  the  Site  Detennination  Tool. 


Begin  by  requesting  the  interviewee  to  complete  the  Informed  Consent  and  Privacy  Act  forms. 

I.  Purpose  of  Interview  (2  minutes) 

Provide  the  interviewee  with  a  background  on  why  we’re  conducting  interviews  and  what  we 
want  to  ask  about: 

We  are  interviewing  you  today  to  understand  what  you  consider  when  you  think  about 
whether  a  task  should  be  trained  in  an  institutional  setting,  at  home  station,  or  via 
individual  study.  The  Army  has  asked  us  to  define  the  variables  that  make  a  task 
enduring,  and  thus  well  suited  to  institutional  training,  vice  non-enduring,  and  hence 
more  appropriate  for  unit  training.  By  “enduring,”  we  mean  not  specific  to  and  lasting 
longer  than  the  current  war  or  theater,  or  not  subject  to  change  across  AOs  within  a 
particular  theater.  Once  we  have  identified  what  makes  a  task  enduring  or  not,  we  will 
work  with  the  Institute  for  NCO  Professional  Development  to  create  a  tool  that  helps 
training  developers,  task  analysts,  CTSSB  members,  or  others  better  identify  where 
training  should  be  placed,  OR  more  easily  justify  their  task  placement  recommendations 
to  leadership. 

Do  you  have  any  questions  about  why  we’re  talking  to  you  today? 

We  would  like  your  permission  to  record  this  interview.  We  will  ensure  that  the 
recording  will  only  be  used  within  our  project  team,  and  that  none  of  your  comments  will 
ever  be  attributed  to  you.  Do  we  have  your  pennission?  [If  so,  begin  recording  now.] 

II.  SME  Background  and  Experience  (5  minutes) 

Collect  demographic  infonnation  from  the  interviewee.  Complete  the  Demographics  Fonn  to 
record  his  or  her  responses. 
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Before  we  start  asking  you  about  enduring  and  non-enduring  tasks,  please  tell  us  a  little  bit 
about  your  background  and  experience. 

[for  Active  Duty  personnel] 

•  How  long  have  you  been  in  the  Army? 

•  Do  you  have  prior  experience  in  any  other  military  service? 

•  What  is  your  current  rank? 

•  What  is  your  current  duty  position,  and  how  long  have  you  been  in  it? 

•  What  past  positions  have  you  held  as  an  instructor? 

•  What  past  positions  have  you  held  as  a  training  developer  or  task  analyst? 

•  Have  you  had  any  other  training-related  job  experience? 

•  How  many  deployments  have  you  had?  For  each,  please  tell  me  the  year(s)  you  were 
deployed,  the  theater  (e.g.,  Iraq  or  Afghanistan),  the  length  of  the  deployment,  and 
your  duty  position  in  theater. 

[for  GS/contractor  personnel] 

•  Were  you  formerly  on  active  duty?  If  so,  how  long  were  you  in  the  Army? 

•  Do  you  have  prior  experience  in  any  other  military  service? 

•  What  was  your  rank  when  you  retired/got  out? 

•  What  is  your  current  job,  and  how  long  have  you  been  in  it? 

•  What  past  positions  have  you  held  as  an  instructor? 

•  What  past  positions  have  you  held  as  a  training  developer  or  task  analyst? 

•  Have  you  had  any  other  training-related  job  experience? 

•  Do  you  have  experience  as  a  participant  or  outside  observer  of  a  CTSSB? 

•  Have  you  been  deployed?  If  so,  how  many  deployments  have  you  had?  For  each, 
please  tell  me  the  year(s)  you  were  deployed,  the  theater  (e.g.,  Iraq  or  Afghanistan), 
the  length  of  the  deployment,  and  your  duty  position  in  theater. 

III.  Site  Selection  (20  minutes) 

Collect  information  about  what  factors  or  characteristics  should  be  considered  when  selecting  the 
training  site  for  a  task. 

What  kinds  of  tasks  should  be  trained  in  an  institutional  setting? 

•  What  makes  a  task  appropriate  for  the  NCOAs? 

•  How  do  you  know  that  a  task  is  best  suited  for  the  NCOAs? 

•  What  are  some  examples  of  tasks  taught  in  the  NCOA  here? 

•  Why  are  those  tasks  taught  here? 

•  Which  of  those  tasks  should  not  be  taught  here,  and  why? 

•  Which  of  these  tasks  would  you  consider  to  be  “enduring”? 

•  When  a  task  is  enduring,  what  other  characteristics  does  it  have? 

•  How  would  you  predict  whether  a  new  task  is  likely  to  be  enduring? 
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•  For  the  ones  that  are  not  enduring  but  taught  here  in  the  NCOA,  what  about  the  task 
makes  it  fit  here? 

•  For  the  characteristics  you’ve  talked  about  for  tasks  to  be  trained  in  the  institution 
[name  the  characteristics],  which  are  the  highest  drivers? 

What  kinds  of  tasks  should  be  trained  at  home  station? 

•  What  makes  a  task  appropriate  for  home  station? 

•  How  do  you  know  that  a  task  is  best  suited  for  home  station? 

•  What  are  some  examples  of  tasks  taught  at  home  station? 

•  Why  are  those  tasks  taught  there? 

•  Which  of  those  tasks  should  not  be  taught  there,  and  why? 

•  Which  of  these  tasks  would  you  consider  to  be  “enduring”? 

•  Which  would  you  consider  to  be  “non-enduring”? 

•  When  a  task  is  non-enduring  or  dynamic,  what  other  characteristics  does  it  have? 

•  How  would  you  predict  whether  a  new  task  is  likely  to  be  non-enduring/dynamic? 

•  For  the  ones  that  are  enduring  but  taught  at  home  station,  what  about  the  task  makes  it 
fit  there? 

•  For  the  characteristics  you’ve  talked  about  for  tasks  to  be  trained  at  home  station 
[name  the  characteristics],  which  are  the  highest  drivers? 

What  kinds  of  tasks  should  be  trained  through  self-study? 

•  What  makes  a  task  appropriate  for  self-study? 

•  How  do  you  know  that  a  task  is  best  suited  for  self-study? 

•  What  are  some  examples  of  tasks  taught  via  self-study? 

•  Why  are  those  tasks  taught  as  such? 

•  Which  of  those  tasks  should  not  be  taught  that  way,  and  why? 

•  Which  of  these  tasks  would  you  consider  to  be  “enduring”? 

•  For  the  ones  that  are  enduring  but  taught  via  self-study,  what  about  the  task  makes  it 
fit  there? 

•  For  the  characteristics  you’ve  talked  about  for  tasks  to  be  trained  via  self-study  [name 
the  characteristics],  which  are  the  highest  drivers? 

What  is  challenging  about  figuring  out  where  a  task  should  be  trained? 

•  What  mistakes  have  you  seen  made,  where  tasks  are  inappropriately  placed  for 
training? 

•  Can  you  give  some  examples  of  tasks  that  were  slotted  in  the  wrong  place? 

•  When  tasks  are  slotted  inappropriately,  what  are  some  of  the  main  reasons  for  failure? 

•  How  often  do  you  believe  that  tasks  are  slotted  for  training  in  the  right  place? 

•  When  tasks  are  slotted  appropriately,  what  are  the  reasons  for  success? 

•  What  elements  of  the  process,  if  any,  interfere  with  properly  placing  tasks? 

•  Who  are  the  players  and  decision  makers  in  the  process? 
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•  How  are  tasks  considered  differently  now,  during  wartime,  than  they  were  prior  to  the 
wars  in  Afghanistan  and  Iraq? 

•  What  factors  should  be  considered,  but  are  not,  when  selecting  training  sites? 

•  How  do  you,  personally,  think  through  task  placement  now  that’s  different  from  how 
you  used  to  think  about  it?  What  have  you  gotten  smarter  about? 

•  If  the  task  selection  board  were  comprised  of  E3s  and  E4s,  what  mistakes  would  you 
expect  them  to  make  in  recommending  training  sites  for  certain  tasks?  What  would 
they  fail  to  consider  in  selecting  sites? 

What  is  the  impact  of  the  ALC  2015  on  the  way  CTSSBs  and  proponents  select  sites  for 
training? 

IV.  Enduring  vs.  Non-Enduring  Tasks  (20  minutes) 

Collect  information  regarding  how  the  interviewee  thinks  about  enduring  and  non-enduring 
tasks. 

Are  you  familiar  with  the  terms  “enduring”  and  “non-enduring”? 

•  What  other  terms  have  you  heard  in  common  usage  that  mean  something  similar? 

•  What  about  a  task  makes  it  enduring  [or  use  SME’s  term]  or  not? 

•  How  do  you,  personally,  judge  whether  or  not  a  task  is  enduring? 

•  What  information  would  help  you  do  a  better  job  of  judging  a  task  as  enduring  or 
non-enduring? 

•  Do  you  currently  consider  task  endurability  as  a  factor  when  you  place/recommend 
sites  for  task  training? 

•  What  kind  of  knowledge  or  experience  does  a  person  need  to  do  a  good  job  of 
identifying  whether  or  not  a  task  is  enduring? 

•  What  does  doctrine  stipulate  about  what  makes  a  task  enduring  or  non-enduring? 

Please  look  at  this  list  of  tasks  in  answering  the  following  questions  [provide  task  list  from 
current  CTSSB]. 

•  Of  all  the  tasks  on  the  list,  which  three  are  the  best  suited  for  institutional  training? 
Why? 

•  Which  three  are  worst  suited  for  institutional  training?  Why? 

•  Which  three  are  the  most  likely  to  be  enduring?  Why? 

•  Which  three  are  the  least  likely  to  be  enduring?  Why? 

•  Which  three  are  the  most  difficult  to  judge  as  enduring  or  non-enduring?  In  other 

words,  which  do  you  see  as  the  most  prone  to  be  misjudged?  Why? 

•  What  information  would  help  you  detennine  if  it  is  an  enduring  or  non-enduring  task? 

You  have  identified  the  following  factors  or  characteristics  as  ones  that  make  a  task 
enduring/non-enduring.  [Say  back  the  factors/characteristics  from  interviewer  notes.] 
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•  Which  of  these  factors  are  easy  to  identify  for  a  member  of  a  CTSSB? 

•  Which  of  these  factors  are  difficult  to  identify? 

•  Which  of  these  factors  are  difficult  to  measure? 

•  What  are  some  other  means  by  which  someone  could  detennine  whether  or  not  a 
factor/characteristic  is  present  in  a  task?  Are  there  other  indicators  he/she  could  use? 

•  Does  operational  experience  impact  a  person’s  ability  to  identify  these  factors  and 
characteristics  in  a  task?  How  so? 

•  Does  training/instructor  experience  impact  a  person’s  ability  to  identify  these  factors 
and  characteristics  in  a  task?  How  so? 

•  Is  there  any  other  type  of  experience  that  would  improve  a  person’s  ability  to  identify 
these  factors?  How  so? 


V.  Site  Selection  Process  -  Entry  Points  to  Support  (10  minutes) 

Collect  information  about  who  are  the  appropriate  users  of  a  Site  Detennination  Tool,  and  where 
in  the  process  it  would  be  optimally  implemented. 

We’ve  been  talking  about  what  makes  a  task  enduring  or  non-enduring,  and  what  are  the 
other  factors  that  detennine  where  a  task  should  ideally  be  trained.  We  discussed  at  the 
beginning  that  our  goal  is  to  develop  a  tool  that  helps  people  determine  the  most  appropriate 
site  for  task  training. 

•  Who  is  most  in  need  of  assistance  to  identify  where  a  task  is  best  trained? 

•  Should  this  tool  focus  on  helping  people  make  recommendations,  or  helping  them 
explain  or  justify  their  recommendations  to  leadership,  or  something  else? 

•  Where  and  when  in  the  process  would  you  see  this  tool  being  used? 

•  Should  the  tool  be  web-based?  Should  it  tie  into  other  technologies? 

•  What  are  the  most  important  functions  the  tool  should  support  or  perform? 


VI.  Wrap  Up 

What  questions  should  we  have  asked  that  we  didn’t  ask? 
What  questions  do  you  have  for  us? 
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Appendix  D.  Site  Selection  Tool 

Question  Definitions 

Site  Selection  Tool  Questions 

Purpose 

Definition 

DURING  TRAINING, 

„  _  what  is  the  risk  of  injury 

„  to  personnel  or  damage  to 

concern .  . 

equipment  when  training 

this  task? 

When  there  is  a  safety  concern 
associated  with  the  task,  it  requires 
face-to-face  training  and  is  not 
appropriate  for  self-development. 

The  danger  associated  with  training  or  performing  the  task. 

Significant  risk  (life,  limb,  eyesight)  may  be  linked  with  incorrect 
performance,  either  in  training  or  operational  contexts.  This 
question  is  about  the  degree  of  risk  associated  with  conducting  the 
task. 

Is  actual  equipment  -  a 

Equipment  weapon,  vehicle,  or 

and  hands-on  system  -  and  hands-on 

training  training,  instead  of  an 

required?  equipment  substitute, 

required  to  train  this  task? 

When  a  task  requires  that  equipment 
be  present  during  training  so  that  a 
trainee  can  physically  manipulate  it, 
self-development  is  generally 
inappropriate. 

The  extent  to  which  task  performance  and  training  requires  motor  or 
hands-on  performance,  or  physical  manipulation  of  tools, 
equipment,  or  items.  This  question  is  about  whether  the  training 
must  employ  the  actual  equipment  that  would  be  used  operationally, 
or  whether  computer  based  training,  a  simulation,  or  a  classroom- 
based  substitute  can  be  used  instead. 

Difficult  to  How  complex  or  difficult 
learn?  to  leam  is  this  task? 

Tasks  that  are  higher  in  complexity 
or  difficultly  are  less  suited  for  self¬ 
development  and  more  suited  for 
training  by  an  instructor  or  SME. 

The  effort  required  to  leam  the  task,  usually  based  on  the  degree  of 
difficulty  of  the  concepts  or  procedures  involved.  Tasks  that  are 
more  complex  require  a  human  trainer  for  instruction,  to  provide 
examples,  and  to  answer  questions.  This  question  is  about  the  degree 
of  task  complexity. 

Assess  Does  a  trainer  need  to 

performance  .  ,  e 

\  ,  ,  .  watch  performance  tace- 

y  wa  c  mg  to_face  to  assess  it? 

tace-to-iace? 

Tasks  that  require  a  trainer  to 
observe  the  trainee's  performance  in 
order  to  assess  it  are  not  appropriate 
for  self-development. 

The  extent  to  which  a  Soldier’s  performance  must  be  observed  in 
order  to  evaluate  it  against  the  standard.  This  question  refers  to 
whether  a  trainer  is  required  to  watch  NCOs  perform  the  task,  in 
person,  in  order  to  conduct  an  accurate  assessment. 

Will  the  training  change 
needed  more  .,  o 

,  „  more  than  once  a  year.’ 

than  yearly? 

When  the  training  is  subject  to 
frequent  change,  self-development 
may  be  inappropriate  due  to  the 
expense  of  modifying  computer- 
based  or  distributed  learning  (dL) 
training  content. 

The  likelihood  that  task  standards  will  change  and  instruction  will 
require  modification  to  keep  the  pace  with  the  evolving  conditions. 

If  the  training  content  will  change  more  than  once  a  year,  the  task  is 
probably  not  a  good  fit  for  self-development. 

SMEs  .  ... 

, ,  Are  SMEs  readily 

avadable  or  ,  ,  .,  ,  , 

,  available  or  accessible  to 

accessi  e  a  trajn  this  task  at  the  unit? 

unit? 

If  task  expertise  is  not  accessible  at 
the  unit,  the  task  is  more  likely  to 
require  institutional  training. 

The  extent  to  which  trainers,  small  unit  leaders,  or  subject  matter 
experts  are  training  the  task  at  the  unit.  This  question  refers  to 
whether  subject  matter  expertise  typically  exists  at  the  unit,  and  also 
whether  those  SMEs  are  readily  available  to  NCOs  for  training. 

Annual  Is  recurring  qualification 

qual./cert.  at  or  certification  conducted 

unit?  at  unit? 

When  units  require  annual 
qualification  or  certification,  a 
program  of  training  likely  already 
resides  at  the  unit. 

Whether  or  not  there  is  an  annual  training  or  qualification 
requirement  that  is  satisfied  at  the  unit.  The  rationale  behind  this 
question  is  that  when  annual  testing  occurs  at  the  unit,  there  is  a 
strong  likelihood  that  NCOs  receive  the  required  training  at  the  unit. 
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Site  Selection  Tool  Questions 

Purpose 

Definition 

Is  the  task  universal  and 

task,  umiorm  v  . 

■  .  .  requiring  unitormity  ot 

training  2  .  « 

.  ”  training  across  the  MOS; 

required? 

When  the  task  is  relevant  to  all 
individuals  in  the  MOS  and  also 
requires  that  all  Soldiers  be 
instructed  on  it  in  a  standardized 
manner,  then  it  is  appropriate  to  be 
trained  at  the  institution. 

This  question  has  two  parts.  Universality  is  about  the  extent  to 
which  the  task  is  relevant  to  NCOs  across  the  Force  or  across  the 
MOS,  regardless  of  job  assignment.  Standardization  refers  to  how 
important  it  is  that  all  Soldiers  leam  to  conduct  the  task  using  the 
same  procedure.  This  question  asks  whether  most  NCOs  will 
employ  the  task  on  the  job,  and  whether  they  must  know  or  use  the 
book  standard  to  be  effective. 

Reliable  and  Is  a  reliable  and  controlled 
controlled  assessment  highly 

assessment?  important? 

Tasks  that  require  extra  assurance 
that  Soldiers  have  grasped  the 
important  concepts  and  performance 
steps  are  better  suited  for 
institutional  training,  where  a  reliable 
assessment  is  more  likely  to  always 
take  place. 

The  importance  associated  with  ensuring  task  training  has  been 
delivered  and  performance  has  been  assessed  for  each  and  every 
NCO.  Some  tasks  require  an  extra  degree  of  assurance  that  every 
Soldier  who  has  gone  through  the  training  actually  grasps  the  task 
and  can  perform  it  effectively.  This  question  refers  to  the  criticality 
that  an  instructor  be  present  to  ensure  each  and  every  NCO  is  trained 
to  standard. 

Peer-to-peer  Is  sharing  experiences 

learning  with  peers  from  other 

critical  and  units  critical  and  required 

required?  to  maximize  learning? 

When  trainees  gain  a  significant 
benefit  from  hearing  about  peers' 
experiences  in  a  range  of  other 
mission  or  garrison  contexts, 
institutional  training  is  often  more 
appropriate. 

The  degree  to  which  there  is  a  learning  advantage  associated  with 
exposure  to  the  experiences  of  Soldiers  from  other  units  or 
backgrounds.  When  a  task,  such  as  a  combat  tactic,  must  be 
adjusted  for  execution  in  different  mission  types  or  operational 
environments,  there  is  great  learning  value  in  hearing  examples  of 
the  many  ways  to  conduct  the  task.  This  question  refers  to  whether 
there  is  a  particular  benefit  from  exposure  to  peers  from  different 
units  who  can  speak  to  how  mission  or  environmental  factors  impact 
task  execution. 

Criticality  to  How  critical  is  the  task  to 
leadership?  the  leadership  role? 

Tasks  with  a  heavy  leadership 
component  may  be  most  amenable  to 
an  institutional  course. 

The  importance  or  centrality  of  the  task  for  being  an  effective 
leader.  This  question  is  about  whether  or  not  the  task  is  one  of  the 
core  functions  of  a  leader. 

„  -  ,  ,  .  Do  most  units  have  the 

Safety/equip  e  .  , 

J  satety  measures  and 

ment  .  . 

....  ,  equipment  readily 

available  at  .  f 

„  available  to  pertorm 

Umt?  training? 

When  the  other  factors  indicate  unit 
training  is  more  appropriate  AND  the 
unit  has  the  required  equipment  and 
safety  measures  to  conduct  the 
training,  then  unit  training  is 
probably  optimal. 

Whether  the  necessary  equipment  or  safety  measures  for  training  the 
task  are  present  and  available  at  most  units.  This  question  provides 
a  check  to  ensure  that  if  a  task  is  recommended  for  training 
exclusively  at  the  unit,  most  units  across  the  Army  will  have  the 
means  to  conduct  that  training. 
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Appendix  E.  Site  Selection  Tool  Instructions 

SniSlUCTWH  root  User  Quick  Reference  Guide 


This  guide  provides  basic  instructions  for  using  the  Site  Selection  Tool  (SST). 

1.  Click  Enable  Content  2.  Read  Task  Title  3.  Answer  Questions  For  Task 


One*  the  excel  Hie  <s  opened,  a  yetow  Meeeag#  Bar 
appears  with  a  shield  icon  and  th*  Enable  Content  or 
Enable  Editing  button  On  the  Message  Bar  elk* 
Enable  Content  or  Enable  Editing 


You  will  be  rovewng  the  tasks  fcsted  to  rriako  a  site 
recommendation.  Read  each  task  in  the  column  labeled 
Task  Title  if  you  would  Ike  to  indude  the  criticality  rating 
for  each  task,  (1)  select  columns  B  and  0.  \2  i  using  your 
mouse,  right-dek  over  the  two  highlighted  columns,  and  (3) 
select  Unhide  to  reveal  the  hidden  task  criticality  column 


For  each  task,  answer  questions  banning  with  #i 
Your  cursor  will  move  to  the  next  relevant  question 
with  some  questions  greyed  out  Pi*ase  answer  the 
questions  in  the  order  directed  by  the  tool  To  see 

the  complete  question,  hold  your  mouse  pointer  over 
the  red  triangle  at  the  top  of  the  column 


4.  Use  Drop  Down  Menu  or 
Keystrokes 


Use  the  drop  down  menu  on  the  sido  of  the 
highlighted  box  lo  select  your  answer  to  th«  question 
Some  questions  require  a  Yes  or  Mo  Others  require  a 
High  Medum  or  Low.  You  may  also  key  in  Yes.  No 
High.  MefliLm,  or  Low  to  respond  to  each  question 
payng  attention  to  capitafczaticn. 


5.  Reset  Responses  if  Necessary 


Cick  Reset  to  change 
answer 


To  edt  oarler  responses,  please  use  the  Reset  button  to 
the  right  of  the  quesbons  You  may  reset  at  any  time  during 
yoir  review  Once  the  quesbons  are  answered,  a  site 
recommendation  will  automabcaly  appear  under  the  SST 
rocommendabon  column.  Do  not  change  the  system's 
recommendations  and  do  not  answer  questions  to  get  a 
specific  recommendation.  Your  personal  recommendations 
may  be  entered  in  the  next  step. 


6.  Indicate  Recommendation  and 
Make  Comments 


If  you  disagree  with  the  SST  recommendation,  please  use 
the  drop  down  menu  to  select  your  personal  site 
recommendation  If  you  have  additional  comments,  enter 
them  in  the  Comments  column  When  you  have  completed 
the  row,  move  to  the  first  question  for  the  next  task  on  the 
1st  Save  frequently  to  avoid  data  loss. 
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Site  Selection  Tool  Instructions  for  Administrators 

PART  ONE:  SET  UP  THE  FILE 

1 .  Create  a  new  folder  for  your  set  of  tasks  on  your  desktop. 

2.  Copy  over  the  Admin  and  Board  Member  files  to  the  new  folder  you  created  or  use  the  Save  As 
command  to  locate  and  place  the  Admin  and  Board  Member  files  in  the  new  folder. 

a.  Note  Do  not  rename  the  file.  Please  keep  the  same  file  name  when  placing  the  files  in 
the  new  folder. 

3.  Open  the  file  that  contains  your  task  list. 

a.  In  preparation  for  importing  the  task  list,  use  two  columns:  one  for  the  task  number  and 
one  for  the  task  title. 

i.  Note  To  simplify  board  member  inputs,  organize  the  task  list  in  a  logical  manner 
before  importing  it  into  the  SST. 

b.  If  desired,  you  may  also  import  the  task  criticality  rating  for  each  task. 

4.  Open  the  Admin  and  Board  Member  files. 

a.  When  you  open  an  SST  file,  a  bar  at  the  top  of  the  window  will  appear.  Click  Options 
and  select  Enable  this  Content  to  proceed. 


b.  If  you  would  like  to  include  task  criticality  ratings,  in  the  Board  Member  file  (1)  select 
columns  B  and  D,  (2)  using  your  mouse,  right-click  over  the  two  highlighted  columns, 
and  (3)  select  Unhide  to  reveal  the  hidden  task  criticality  column.  The  task  criticality 
column  in  the  Admin  file  is  not  hidden  but  it  is  still  optional  to  include  task  criticality 
ratings. 

c.  Go  to  the  task  list  you  wish  to  paste  into  the  SST.  Highlight  all  the  task  numbers  and  task 
titles  (and  criticality  ratings,  if  applicable)  by  left-clicking  the  first  task  number  and 
dragging  down  diagonally  to  the  last  task  title  (or  the  last  criticality  rating;  this  assumes 
there  are  no  other  columns  between  the  task  number,  task  title,  and  criticality  rating 
columns).  Right-click  the  highlighted  columns  and  select  Copy. 

^  Copy 

d.  Once  you  copy  the  task  list,  right-click  cell  A2  of  the  Board  Member  file  and  select  the 
following: 

i.  Paste  Special  -  Values 
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5.  Once  you  paste  the  tasks  in  the  Board  Member  file,  follow  steps  4b,  4c,  and  4d  to  paste  the  same 
task  numbers  and  titles  to  the  Admin  file. 

6.  When  the  task  numbers  and  titles  have  been  pasted  into  both  the  Admin  and  the  Board  Member 
files,  replace  ‘<insert  course  name  here>’  in  cell  B1  of  both  files  with  the  name  of  the  course  for 
which  the  CTSSB  is  being  conducted. 
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7.  Once  the  course  name  has  been  entered  into  both  the  Admin  and  the  Board  Member  files,  save 
and  close  the  Admin  file. 

a.  Note  You  will  get  a  Privacy  Warning  window.  This  is  normal.  Click  OK  to  continue. 

8.  In  the  Board  Member  file,  click  the  Setup  Sheet  button  in  the  top  left.  Depending  on  the  number 
of  tasks  you  have  entered,  the  program  may  take  a  few  seconds  to  populate  the  Reset  buttons  and 
prepare  the  file  for  distribution. 
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PART  TWO:  DISTRIBUTE  THE  FILES 

Please  Note: 

•  You  can  either  distribute  the  files  individually  via  email  or  by  placing  them  in  an  accessible 
folder  for  the  board  members  to  access  on  their  own. 

•  The  administrator  has  two  options  for  duplicating  the  Board  Member  file: 

1 .  Send  out  or  upload  a  general  Board  Member  file  and  require  each  member  to  enter  his  or 
her  name  behind  both  the  file  name  (e.g.  BoardMember  Brown)  and  the  workbook 
name,  making  sure  to  keep  ‘SST’  in  the  workbook  name  (e.g.  SST  Brown). 

■  Note  We  recommend  using  only  spaces  and  underscores  to  separate  words. 

Avoid  special  characters  (!  ?  @  etc.). 

2.  Create  multiple  files,  making  individual  Board  Member  files  following  the  naming 
conventions  above,  and  send  them  out  or  upload  them  individually. 

•  While  this  naming  convention  is  optional,  it  enables  the  administrator  to  have  complete 
traceability  and  a  better  overview  of  the  responses  by  his  or  her  board  members. 
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PART  THREE:  COMPILE  THE  FILES 

1 .  Once  you  receive  all  Board  Member  files,  place  them  in  the  same  folder  you  created.  Make  sure 
that  just  the  one  Admin  file  and  all  completed  Board  Member  files  are  the  only  existing  files  in 
that  folder.  If  there  are  other  files  in  the  folder,  please  remove  them. 

a.  Note  Do  not  rename  the  Admin  file  until  all  files  have  been  compiled. 

Name 

Admin 

1^*1  BoardMember_Ayers 
^  BoardMember_Brown 
®!|  BoardMember_Erickson 
1^1*1  BoardMemberJackson 
BoardMemberJones 
^*1  BoardMember_McEntire 

2.  Open  each  Board  Member  file  and  click  the  Setup  Sheet  button  to  link  it  to  the  Admin  file.  Save  it 
and  close. 

3.  Open  the  Admin  file. 

4.  Click  ‘Options’  button  to  open  the  security  warning. 


^  Security  Warning  Macros  have  been  disabled  Options.. 


5.  Select  ‘Enable  this  Content’  to  enable  macro. 


Help  protect  me  from  unknown  content  (recommended) 


Enable  this  content! 
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6.  To  calculate,  first,  use  the  drop  down  menu  next  to  Set  %  Consensus  to  choose  if  the  program 
should  calculate  agreement  at  50%,  60%,  70%,  or  80% 

7.  Next,  click  the  Load  Board  Sheets  button  at  the  top  left  of  the  window.  The  program  will  open 
and  import  each  board  member  sheet  in  the  folder. 


A  “J* 


Insert  Page  Layout 


Review  View  Developer  Acrobat 


Admin  -  Microsoft  Excel 


Task  Task  Self  %  Institution/  Self 

Number _ Task  Title _ Criticality  Institution  Unit  Study  Consensus  Institution  Functional  Unit  Study 


3 

71-170-050 

APPLY  MESSAGE  ADDRESSING 

FEATURES  IN  FDCB2  -  DFT  VERSION  6.4 

13-573-080 

Conduct  Operations  Security  (OPSEC) 

8.  Finally,  return  to  the  ‘Compiled  SST’  worksheet  and  click  the  Compile  button  at  the  top  left  of 
the  window. 

9.  The  program  has  now  compiled  all  the  Board  Member  files  into  the  Admin  file  and  calculated  the 
concurrence. 
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SITE  SELECTION  TOOL  ASSESSMENT  SURVEY 

Please  indicate  on  the  scale  from  1-5  your  level  of  agreement  or  disagreement  with  the  following 
statements.  If  the  question  is  not  applicable  to  you,  please  leave  it  blank. 


Strongly  Strongly 

Disagree  Agree 


1.  The  Site  Selection  Tool  was  valuable  to  tne.  1  2  3  4  5 

2.  The  questions  in  the  Site  Selection  Tool  were  easy  to  ^  2  3  4  5 

understand. 


3.  The  questions  in  the  Site  Selection  Tool  were  easy  to 
answer. 


2  3  4  5 


4.  The  Site  Selection  Tool  was  simple  to  use 


2  3  4  5 


5. 1  was  able  to  be  more  objective  with  my  site 
recommendations  using  the  Site  Selection  Tool. 

6. 1  was  able  to  produce  better  site  recommendations 
using  the  Site  Selection  Tool. 

7.  The  Site  Selection  Tool  helped  me  to  consider 
factors  I  would  not  have  otherwise  thought  about 
when  recommending  a  training  site. 

8.  The  training  site  recommendation  generated  by  the 
Site  Selection  Tool  typically  met  my  expectations. 

9.  The  Site  Selection  Tool  is  reliable. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


10.  The  Site  Selection  Tool  was  accurate  in 
recommending  the  correct  site  for  each  task. 

11.  Using  the  Site  Selection  Tool  was  NOT  time 
consuming. 

12.1  would  recommend  the  Site  Selection  Tool  for 
future  CTSSBs. 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


13.  All  components  of  the  Site  Selection  Tool 
functioned  properly. 


2  3  4  5 


CONTINUED  ON  THE  BACK 
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Please  read  and  answer  the  following  questions. 

A.  What  features  of  the  Site  Selection  Tool  did  you  find  to  be  the  most  valuable? 


B.  What  elements  of  the  Site  Selection  Tool  did  you  find  to  be  irrelevant  or  distracting? 


C.  What  are  the  three  most  important  changes  that  should  be  made  to  the  Site  Selection  Tool? 


D.  What  changes  would  you  make  to  the  way  the  Site  Selection  Tool  was  used  as  part  of  the 
CTSSB  process? 
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Appendix  G.  Focus  Group  Protocol 


INTERVIEW  PROTOCOL 

Front-End  Analysis  for  the  NCOES 
Site  Selection  Tool  Assessment  Protocol 


Objectives 

1 .  Gather  user  feedback  regarding  the  relevance,  ease  of  use,  and  value  of  the  Site  Selection 
Tool. 

2.  Identify  the  face  validity  of  the  factors  included  in  the  tool. 

3.  Identify  revisions  and  improvements  necessary  for  improved  SST  functionality. 

4.  Determine  whether  the  tool  can  be  integrated  with  existing  critical  task  review  process. 


Begin  by  requesting  the  interviewee  to  complete  the  Infonned  Consent  and  Privacy  Act  forms. 

I.  Purpose  of  Interview  (2  minutes) 

Provide  the  interviewee  with  a  background  on  why  we’re  conducting  interviews  and  what  we 
want  to  ask  about: 

We  are  interviewing  you  today  to  identify  any  information  that  would  help  us  improve  the  FEA 
Site  Selection  Tool  that  you  just  finished  using  during  the  CTSSB  process.  We  created  this  tool 
to  assist  CTSSB  members,  course  directors,  instructors,  or  others  better  identify  where  training 
should  be  placed,  OR  more  easily  justify  their  task  placement  recommendations  to  leadership. 

Do  you  have  any  questions  about  why  we’re  talking  to  you  today? 

We  would  like  your  permission  to  record  this  interview.  We  will  ensure  that  the  recording  will 
only  be  used  within  our  project  team,  and  that  none  of  your  comments  will  ever  be  attributed  to 
you.  Do  we  have  your  permission?  [If  so,  begin  recording  now.] 

II.  SME  Background  and  Experience  (3  minutes) 

Collect  demographic  information  from  the  interviewee.  Demographic  information  will  be  used 
only  to  consider  the  participants  experience  with  Critical  Task  and  Site  Selection  Boards. 

Before  we  start  asking  you  about  the  tool,  please  tell  us  a  little  bit  about  your  background. 

•  How  long  have  you  been  in  the  Army? 

•  What  is  your  current  rank? 

•  How  many  CTSSBs  have  you  participated  in  during  your  service? 

•  What  is  your  current  duty  position,  and  how  long  have  you  been  in  it? 

•  How  many  deployments  have  you  had?  For  each,  please  tell  me  the  year(s)  you  were 
deployed,  the  theater  (e.g.,  Iraq  or  Afghanistan),  the  length  of  the  deployment,  and  your  duty 
position  in  theater. 
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III.  Site  Selection  Tool  Impressions  (20  minutes) 


Collect  information  about  his/her  experience  using  the  tool  and  the  overall  effectiveness  for 
improving  the  current  CTSSB  site  selection  process. 

What  is  the  overall  value  of  the  Site  Selection  Tool? 

•  Did  you  like  using  the  SST? 

•  Did  the  tool  make  your  job  easier? 

•  Do  you  think  the  tool  helped  lead  to  better  training  site  recommendations  than  if  the 
recommendation  were  made  without  the  tool? 

What  were  your  impressions  of  the  SST’s  interface  and  functionality? 

•  Were  you  able  to  understand  how  to  interact  with  the  SST?  What  did  and  did  not  make  sense? 

•  Can  you  describe  how  the  tool  helped  you?  If  it  did  not,  why? 

•  To  what  extent  did  the  SST  make  a  difference  in  the  way  you  recommend  training  sites? 

•  Did  the  graying  out  of  the  questions  make  sense  to  you?  Why  or  why  not? 

•  How  often  did  you  write  a  comment  about  the  task? 

•  Were  you  happy  with  the  requirement  to  agree  or  disagree? 

•  In  what  ways  did  you  find  the  tool  to  be  effective?  Ineffective? 

Did  you  find  the  questions  asked  in  the  SST  to  make  sense  and  have  relevance  to  your  job  of 
recommending  training  sites  for  tasks? 

•  How  did  you  interpret  and  use  each  question? 

•  Did  you  have  trouble  answering  with  a  yes  or  a  no?  Would  you  have  rather  rating  each 
question  on  a  scale?  For  which  questions  would  you  want  to  provide  a  rating  rather  than  a 
yes/no? 

•  Did  you  usually  agree  with  the  system’s  site  recommendation?  Why  or  why  not? 

•  What  was  your  impression  of  the  two  questions  regarding  safety  and  risk?  Were  you  able  to 
discriminate  tasks  where  safety  is  a  major  concern  versus  a  minor  concern? 

•  Were  you  surprised  by  any  of  the  questions? 

•  Did  you  find  any  of  the  questions  to  be  ones  that  you  normally  would  not  consider  in  making 
a  site  recommendation? 

o  If  so,  were  these  good  inclusions  or  bad  inclusions? 

How  good  were  the  recommendations  of  the  Site  Selection  Tool? 

•  Can  you  give  some  examples  of  SST  recommendations  that  you  did  not  agree  with?  Why  did 
you  disagree?  Were  there  other  factors  that  should’ve  been  considered? 

•  How  reasonable  were  the  recommendations?  What  did  not  make  sense? 

In  what  ways  can  the  SST  be  integrated  with  the  existing  CTSSB  process? 

•  Did  you  agree  with  the  way  the  tool  supported  the  site  selection  process? 

•  Did  the  SST  support  group  discussions  of  tasks  that  were  not  clear  cut?  Why  or  why  not? 

•  Did  the  SST  make  the  CTSSB  process  more  efficient?  Why  or  why  not? 

•  Did  the  SST  make  the  CTSSB  outcomes  stronger?  More  justifiable?  Why  or  why  not? 
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Did  the  tool  integrate  smoothly  with  the  CTSSB  process? 

How  should  it  be  implemented  or  adapted  to  better  meet  the  board  member’s  needs? 


IV.  Site  Selection  Tool  Improvements  (10  minutes) 

Collect  information  about  any  revisions  that  should  be  considered  for  the  Site  Selection  Tool. 
How  would  you  improve  the  Site  Selection  Tool,  and  why? 

•  What  are  the  important  factors  missing  from  the  tool  that  may  be  necessary? 

•  What  elements  of  the  tool  need  to  be  rearranged? 

•  What  questions  within  the  tool  need  to  be  reworded? 

•  Describe  the  problems  (if  any)  you  encountered  during  the  process? 

What  features  would  you  change  to  better  support  your  thought  processes  during  the  CTSSB? 

•  What  did  you  like/did  not  like  about  the  tool’s  interface? 

•  Were  there  too  many  questions  to  answer  for  each  task  or  too  few? 

•  Would  you  have  preferred  to  answer  the  questions  with  a  rating  as  opposed  to  yes/no 
selections? 

•  What  do  you  recommend  to  change  when  considering  the  appearance  of  the  tool? 


V.  Wrap  Up 

What  questions  should  we  have  asked  that  we  didn’t  ask? 
What  questions  do  you  have  for  us? 
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Appendix  H.  Implementation  Guidance 


USING  THE  SITE  SELECTION  TOOL  IN  THE  CTSSB 


OVERVIEW 

The  Site  Selection  Tool  (SST)  is  a  computer-based  Excel  spreadsheet  designed  to  draw  upon 
the  wisdom  of  site-selection  experts  to  aid  the  novice  site  selector — the  Critical  Task  Site 
Selection  Board  (CTSSB)  member — in  understanding  what  makes  a  critical  task  best  suited  for 
institutional  training,  operational  training,  and  self-study.  It  is  intended  to  augment,  but  not 
replace,  the  judgment  and  experience  of  CTSSB  members.  Based  on  board  member 
responses  to  questions  about  a  task,  it  makes  a  recommendation  for  training  site.  When  users 
disagree  with  the  SST  recommendation,  they  record  their  site  placement  opinion.  The  SST 
compiles  the  results  across  board  members,  capturing  both  SST  recommendations  and  the 
board  members’  opinions.  Tasks  for  which  there  is  a  high  level  of  disagreement  are  flagged  for 
discussion  by  the  board.  The  SST  thus  improves  the  site  recommendations  of  the  CTSSB,  and 
adds  efficiency  to  the  site  selection  process. 

PURPOSE 

The  SST  was  developed  to  aid  CTSSBs  in  the  site-selection  phase  of  the  board  process. 
Specifically,  the  SST  has  two  functions. 

Time  Saving.  The  SST  identifies  tasks  for  which  consensus  regarding  training  site  location  is 
high,  and  thus  discussion  is  unnecessary.  Conversely,  tasks  for  which  consensus  is  low  are 
highlighted  to  indicate  the  tasks  most  in  need  of  discussion.  In  this  way,  the  boards’  time  is 
allocated  to  discussing  only  those  tasks  upon  which  board  members  disagree. 

Site-Selection  Expert  Insight.  The  SST  aids  Military  Occupational  Specialty  (MOS)  subject 
matter  experts  (SMEs) — the  CTSSB  members — in  considering  factors  affecting  training  site 
decisions  that  they  may  not  consider  on  their  own.  These  factors  have  been  identified  by  site- 
selection  SMEs  as  key  considerations  forjudging  a  task’s  suitability  for  training  in  the  institution, 
at  the  unit,  or  via  self-study.  The  SST  does  not  purport  to  “know  better”  than  the  board 
members  regarding  their  MOS  expertise.  Rather,  it  cues  them  about  site  selection  factors,  such 
as  availability  of  resources  at  a  unit,  that  may  not  be  immediately  recognized  as  points  of 
consideration  by  these  first-time  CTSSB  members. 

DEVELOPMENT  &  LOGIC 

The  SST  was  developed  through  a  research  effort  sponsored  by  the  Institute  for  NCO 
Professional  Development  (TRADOC)  and  conducted  by  the  U.S.  Army  Research  Institute 
(ARI),  Cognitive  Performance  Group  (CPG),  and  Dynamics  Research  Corporation  (DRC). 
Through  this  research,  ARI,  CPG  and  DRC  consulted  with  site-selection  SMEs  across  a  range 
of  MOSs  and  NCOES  proponents  to  include  course  managers  and  instructors,  course 
developers,  and  task  analysis  branch  personnel.  CTSSB  members  were  also  interviewed  as 
the  target  user  audience.  The  site-selection  SMEs  identified  the  key  factors  they  consider,  as 
well  as  their  decision  making  process.  These  expert  considerations  were  coalesced  into  the 
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logic  flow  chart  in  Figure  1 .  For  a  more  detailed  description  of  each  question  and  the  rationale 
for  its  inclusion  in  the  SST,  see  Attachment  1 . 


1.  DURING  TRAINING,  what  isthe  risk 
of  injury  to  personnel  or  damage  to 
equipment  when  trainingthistask? 


Low  L 


2.  Is  actual  equipment-a  weapon, 
vehicle,  or  system -and  hands-on 
training,  instead  of  an  equipment 
substitute,  required  to  train  this  task? 


No 


3.  How  complex  ordifficultto  learn  is 
this  task? 


4.  Does  a  trainer  need  to  watch 
performance  face-to-face  to  assess 
it? 


No  ,, 


I  High,  Medium,  Low  Response  Questions 
I  Yes,  No  Response  Questions 


ro.tn.f3rotn  accocc 


5.  Will  the  training  change  more 
than  once  a  year? 


8.  Is  the  task  universal  and  requiring 
uniformity  of  training  across  the 
MOS? 


Yes  . 


9.  Isa  reliable  and  controlled 
assessment  highly  important? 


No 


I  Yes 


N°1  t 


10.  Is  sharing  experiences  with  peers 
from  other  units  critical  and 
required  to  maximize  learning? 


Yes 


12.  Do  most  units  have  the  safety 
measures  and  equipment  readily 
available  to  perform  trainir^? 


11.  How  critical  is  the  task  to  the 
leadership  role? 


High/ 

Med 


Yes 


No 


49 


w 

Institution 

1 

_  ^ 

4 

Figure  1.  Site  selection  tool  flow  chart  logic 


The  flow  chart  illustrates  the  internal  process  guiding  the  functionality  in  the  SST.  For  each  task 
considered,  the  first  question  set — Questions  1-5 — works  to  identify  whether  self-study  or 
distributed  learning  (dL)  is  feasible.  Responding  affirmatively  to  any  of  these  questions,  rules 
out  self-study/d L.  If  self-study  is  ruled  out,  the  next  question  set — Questions  6,  7,  11,  and  12 — 
considers  whether  unit  training  is  preferable  and  feasible  for  the  task.  Affirmative  answers  to 
Questions  6  and  7,  and  a  low  criticality  to  leadership  assessment  for  Question  11,  prompt  a 
potential  suggestion  of  unit  training.  If  the  Question  12  response  indicates  most  units  have  the 
required  safety  measures  and  equipment,  then  unit  training  is  recommended.  When  the  answer 
to  Question  6  or  7  is  a  ‘no,’  then  the  factors  in  Questions  8-1 1  are  considered  to  determine 
between  institution  and  unit  as  the  recommendation. 


Important:  It  is  usually  not  necessary  to  answer  all  questions  to 
produce  a  site  recommendation.  This  does  not  indicate  that  the 
skipped  questions  are  not  important.  Instead,  when  the  flow  chart 
causes  users  to  skip  some  questions,  it  is  because  (1)  the  logic  flow 
indicates  that  the  answer  to  that  question  will  not  produce  additional 
information  that  would  alter  the  recommendation,  or  (2)  the  earlier  or 
remaining  questions  are  more  informative  regarding  the  site 
recommendation  than  the  skipped  question(s).  Users  always  have  the 
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option  to  disagree  with  the  SST’s  recommendation,  and  user  opinions 
are  both  noted  and  counted  in  the  final  SST  compilation. 
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PROCESS 


General.  The  SST  is  intended  to  be  used  during  the  normal  course  of  the  CTSSB.  Many 
CTSSBs  are  conducted  with  two  distinct  phases:  (1)  a  critically  vote  to  reduce  the  total  task 
inventory  to  only  tasks  that  are  deemed  critical,  and  (2)  a  site  selection  phase  in  which  tasks 
appointed  to  the  critical  task  list  are  recommended  for  institution,  unit  or  self-study  as  the 
training  site.  The  SST  is  particularly  well  suited  to  support  this  popular  approach  to  conducting 
CTSSBs.  In  this  approach,  the  individual  board  members  use  the  SST  via  computers,  and  then 
the  Administrator  compiles  all  the  board  members’  responses  (see  companion  manual:  SST 
Instructions  for  Administrators  for  step-by-step  instructions  on  how  to  set  up  and  run  the  SST 
files).  In  the  compilation  phase,  the  Administrator  selects  a  desired  consensus  level,  and  tasks 
that  do  not  reach  that  consensus  level  are  flagged  and  highlighted  in  amber.  These  highlighted 
tasks  are  the  tasks  intended  for  discussion.  Figure  2  below  shows  how  the  SST  is 
recommended  to  fit  into  the  CTSSB  process. 


Post-CTSSB 


Master  CTL  Site 
Recommendations 
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Board  Admin 
Activities  m _ 

Key 
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_ 1 

Products  1 

CTSSB  Member 
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CTL  Site 

Recommendations 


CTL  Site  Rees 
Compiled  via  SST 
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Figure  2.  Recommended  application  of  the  SST  in  the  CTSSB  process 
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Some  boards  will  not  be  able  to  utilize  the  computer  functionality  of  the  SST  due  to  security 
restrictions.  In  these  cases,  the  SME  guidance  embedded  in  the  SST  can  be  applied  using  a 
paper-based  version  of  the  SST.  For  support,  see  the  Paper-Based  Implementation 
Instructions. 

Prior  to  CTSSB.  Before  the  CTSSB  assembles,  take  the  following  steps: 

1.  Obtain  and  review  the  appropriate  version  of  the  SST  Instructions  for  Administrators.  If 
you  are  using  Microsoft  Excel  2007,  use  the  corresponding  instructions.  If  you  are  using 
Microsoft  2010,  a  separate  set  of  instructions  exists.  If  you  find  through  repeated  use  of 
the  SST  that  your  computers  continually  prompt  users  to  “enable  content”  and  you  would 
like  to  turn  this  feature  off,  apply  the  administrator  instructions  with  the  Macros  Enable 
extension.  You  may  also  wish  to  use  the  Macros  Enable  instructions  if,  when  loading 
board  member  files  during  the  compilation  phase,  that  not  all  board  member  files  are 
loaded  into  the  administrator  file. 

2.  Identify  your  site  information  technology  (IT)  personnel  and  work  with  him  or  her  to 
ensure  every  board  member  will  be  able  to  log  into  and  access  the  SST  files  on  the 
computers  designated  for  use.  Ask  for  his  or  her  recommendation  for  distributing  the 
SST  files  to  board  members,  and  then  collecting  the  completed  files  (e.g.,  via  email  or 
other  means). 

3.  Obtain  and  print  out  copies  of  the  SST  Quick  Reference  Guide,  one  for  each  board 
member.  Plan  to  distribute  these  user  instructions  to  each  board  member,  or  leave  them 
at  each  SST  computer  terminal. 

4.  Once  the  critical  task  list  is  produced  by  the  board,  follow  the  appropriate  SST 
Instructions  for  Administrators  to  set  up  for  board  member  SST  inputs. 

Instructions  to  board  member  users.  When  using  the  SST,  the  board  chair  or  the 
CTSSB  administrator  should  prepare  board  members  by  informing  them  of  the 
following: 

•  The  SST  is  designed  to  assist  you  in  considering  relevant  factors  for  your  site 
recommendations. 

•  Beginning  with  the  first  task  on  the  list,  and  Question  1,  work  through  the  series  of 
questions.  Some  responses  may  cause  the  tool  to  skip  some  of  the  questions.  Please 
continue  to  the  next  highlighted  box.  When  you  answer  a  sufficient  number  of  questions 
for  each  task,  the  SST  will  make  a  recommendation. 

•  If  you  make  a  mistake  while  completing  a  task,  you  can  correct  your  response  by  clicking 
the  “Reset"  button  for  that  particular  task.  “Reset”  will  erase  all  responses  for  the  one 
task.  If  you  attempt  to  change  an  answer  without  clicking  “Reset”,  the  SST  will  not  work 
correctly. 

•  When  there  are  special  circumstances  surrounding  a  particular  task,  you  may  disagree 
with  the  recommendation  of  the  SST.  In  these  cases,  please  record  your  personal 
opinion  of  the  most  appropriate  site,  in  addition  to  the  site  recommendation  produced  by 
the  SST,  in  the  personal  recommendation  column  (Column  S). 

•  When  the  SST  produces  a  recommendation  of  “institution”  but  you  believe  the  task  is 
more  appropriate  for  a  functional  course  than  for  the  course  under  review,  select 
“institutional-functional”  in  the  personal  recommendation  column. 

•  Answer  the  questions  in  order,  skipping  only  any  greyed-out  cells  the  tool  indicates  you 
should  skip. 
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•  If  you  are  unfamiliar  with  a  particular  task  and  do  not  feel  comfortable  answering 
questions  about  it,  you  may  skip  that  task  and  move  on  to  the  next  task  in  the  next  row. 
However,  you  must  skip  the  entire  row  and  not  just  individual  questions  within  the  row. 

•  When  all  board  members  have  completed  their  site  recommendations  for  all  tasks,  the 
group  will  reconvene.  When  board  members  disagree  about  site  placement,  the  board 
will  discuss  the  task  and  resolve  the  differences  of  opinion  to  produce  a  final  site 
re  comm  en  d at  ion . 

•  Work  independently.  The  tool  should  represent  your  opinion. 

•  Do  not  try  to  trick  the  system.  If  you  disagree  with  the  SST,  indicate  that  disagreement. 
However,  if  you  try  to  game  the  system  into  producing  your  opinion  as  it’s  response,  this 
will  not  only  defeat  the  purpose  of  applying  your  expertise,  but  it  will  also  introduce  more 
variability  in  responses,  produce  low  consensus  on  the  task,  and  result  in  more  tasks 
and  more  time  required  for  group  discussion. 

•  This  tool  relies  on  your  knowledge  and  understanding  of  the  tasks  and  should  generally 
produce  a  recommendation  that  is  in  keeping  with  your  expertise.  However,  if  it  does 
not,  you  should  indicate  that  disagreement  in  Column  S  where  you  make  your  personal 
recommendation. 

OTHER  USES 

Pre-CTSSB.  The  SST  is  intended  for  use  after  the  selection  of  critical  tasks  has 
occurred.  However,  if  desired,  the  SST  can  be  completed  by  board  members  prior  to 
arriving  at  the  CTSSB.  If  so  desired,  the  tool  with  the  total  task  inventory  (all  tasks  that 
will  be  subject  to  a  criticality  vote)  can  be  emailed  to  board  members  and  completed  at 
home-unit,  assuming  that  the  Soldier  has  access  to  Microsoft  Excel  2007  or  later.  The 
board  members  will  need  to  be  reminded  to  enable  macros.  In  addition,  board 
members  may  have  questions  about  the  tasks’  conditions,  standards,  and  so  forth.  It  is 
recommended  that  board  members  be  provided  with  instructions  on  how  to  access  this 
information  on  their  own. 

Sub-Divide  Groups.  Some  MOSs  experience  greater  diversity  in  experiences  than 
others.  In  some  cases,  there  may  be  large  categories  of  tasks  belonging  to  an  MOS 
that  are  not  practiced  by  many  people  in  that  MOS.  In  such  a  case  (or  for  other  reasons, 
as  well),  board  administrators  or  users  may  feel  that  they  should  opt  out  of  rating  certain 
tasks.  Although  this  is  not  the  standard  approach  outlined  in  TRADOC  Pamphlet  350- 
70-6,  accommodating  this  preference  while  still  utilizing  the  SST  is  feasible. 

To  allow  only  users  with  relevant  experience  to  rate  certain  tasks,  the  normal  SST 
procedures  are  used  with  one  modification:  instead  of  creating  SST  user  files  with  all 
MOS  tasks,  divide  the  tasks  and  users  into  subgroups  according  to  their  relevant 
knowledge  and  experience.  Create  one  folder  and  one  set  of  SST  files  for  each  subset 
of  critical  tasks.  Then,  administer  the  files  to  the  appropriate  subgroup  of  board 
members.  Compile  the  board  member  responses  in  the  same  way,  but  by  subgroup 
rather  than  the  board  as  a  whole. 

If  only  a  few  tasks  are  unfamiliar  to  a  subset  of  the  board  members,  it  will  not  be 
necessary  to  generate  task  subsets  and  board  member  subgroups.  Instead,  encourage 
board  members  to  opt  out  of  recommending  a  site  for  tasks  they  do  not  feel  qualified  to 
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assess,  by  skipping  the  entire  row.  The  SST  will  calculate  votes  on  a  task  and 
percentage  agreement  across  only  the  board  members  who  have  produced  a  site 
recommendation. 
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COMPUTER  USAGE 

Army  posts  vary  in  their  policies  regarding  access  to  computer  labs,  networks,  share 
drives,  and  guest  accounts.  Therefore,  it  is  not  possible  to  advise  regarding  the  optimal 
process  for  distribution  of  files  to  board  members  and  return  of  files  to  the  administrator 
for  compilation.  As  most  board  members  are  likely  not  stationed  at  the  location  where 
the  CTSSB  is  being  conducted,  it  is  common  that  they  will  need  to  secure  guest 
accounts  and  log-ins  for  the  computers.  Depending  on  the  specific  guest  account 
policies,  users  may  be  able  to  access  share  drives  or  the  public  folder  on  a  specific 
computer’s  hard  drive.  During  the  development  of  the  SST,  email  was  used  to  distribute 
and  return  board  member  files.  It  is  also  possible  to  exchange  files  via  writeable  CDs 
(formatted  to  allow  for  rewriting).  The  board  administrator  must  know  how  to  access 
guest  accounts  and  what  access  guest  accounts  will  allow  prior  to  proceeding  with  SST 
use.  And,  the  distribution  and  return  of  files  should  be  checked  prior  to  the  board’s 
convening.  Consultation  with  the  site  information  technology  (IT)  personnel  is 
recommended. 
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ATTACHMENT  1.  QUESTION  DEFINITIONS  AND  RATIONALE 
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Question  # 

Site  Selection  Tool  Questions 

Purpose 

Definition 

1 

Safety 

concern? 

DURING  TRAINING, 
what  is  the  risk  of 
injury  to  personnel  or 
damage  to  equipment 
when  training  this 
task? 

When  there  is  a  safety 
concern  associated  with 
the  task,  it  requires  face- 
to-face  training  and  is  not 
appropriate  for  self- 
study. 

The  danger  associated  with  training  or  performing 
the  task.  Significant  risk  (life,  limb,  eyesight)  may 
be  linked  with  incorrect  performance,  either  in 
training  or  operational  contexts.  This  question  is 
about  the  degree  of  risk  associated  with 
conducting  the  task. 

2 

Equipment 
and  hands- 
on  training 
required? 

Is  actual  equipment  - 
a  weapon,  vehicle,  or 
system  -  and  hands- 
on  training,  instead  of 
an  equipment 
substitute,  required  to 
train  this  task? 

When  a  task  requires 
that  equipment  be 
present  during  training 
so  that  a  trainee  can 
physically  manipulate  it, 
self-study  is  generally 
inappropriate. 

The  extent  to  which  task  performance  and  training 
requires  motor  or  hands-on  performance,  or 
physical  manipulation  of  tools,  equipment,  or 
items.  This  question  is  about  whether  the  training 
must  emplov  the  actual  eauipment  that  would  be 
used  operationally,  or  whether  computer  based 
training,  a  simulation,  ora  classroom-based 
substitute  can  be  used  instead. 

3 

Difficult  to 
learn? 

How  complex  or 
difficult  to  learn  is  this 
task? 

Tasks  that  are  higher  in 
complexity  or  difficultly 
are  less  suited  for  self- 
study  and  more  suited 
for  training  by  an 
instructor  or  SME. 

The  effort  required  to  learn  the  task,  usually  based 
on  the  degree  of  difficulty  of  the  concepts  or 
procedures  involved.  Tasks  that  are  more 
complex  require  a  human  trainer  for  instruction,  to 
provide  examples,  and  to  answer  questions.  This 
question  is  about  the  degree  of  task  complexity. 

4 

Assess 
performanc 
e  by 

watching 

face-to- 

face? 

Does  a  trainer  need  to 
watch  performance 
face-to-face  to  assess 
it? 

Tasks  that  require  a 
trainer  to  observe  the 
trainee's  performance  in 
order  to  assess  it  are  not 
appropriate  for  self- 
study. 

The  extent  to  which  a  Soldier’s  performance  must 
be  observed  in  order  to  evaluate  it  against  the 
standard.  This  question  refers  to  whether  a  trainer 
is  required  to  watch  NCOs  perform  the  task,  in 
person,  in  order  to  conduct  an  accurate 
assessment. 

5 

Updates 
needed 
more  than 
yearly? 

Will  the  training 
change  more  than 
once  a  year? 

When  the  training  is 
subject  to  frequent 
change,  self-study  may 
be  inappropriate  due  to 
the  expense  of  modifying 
computer-based  or 
distributed  learning  (dL) 
training  content. 

The  likelihood  that  task  standards  will  change  and 
instruction  will  require  modification  to  keep  the 
pace  with  the  evolving  conditions.  If  the  training 
content  will  change  more  than  once  a  year,  the 
task  is  probably  not  a  good  fit  for  self-study. 
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Question  # 

Site  Selection  Tool  Questions 

Purpose 

Definition 

6 

SMEs 

available  or 
accessible 
at  unit? 

Are  SMEs  readily 
available  or  accessible 
to  train  this  task  at  the 
unit? 

If  task  expertise  is  not 
accessible  at  the 
operational  unit,  the  task 
is  more  likely  to  require 
institutional  training. 

The  extent  to  which  trainers,  small  unit  leaders,  or 
subject  matter  experts  are  training  the  task  at  the 
unit.  This  question  refers  to  whether  subject 
matter  expertise  typically  exists  at  the  unit,  and 
also  whether  those  SMEs  are  readily  available  to 
NCOs  for  training. 

7 

Annual 
qual./cert.  at 
unit? 

Is  recurring 
qualification  or 
certification  conducted 
at  unit? 

When  operational  units 
require  annual 
qualification  or 
certification,  a  program 
of  training  likely  already 
resides  at  the  unit. 

Whether  or  not  there  is  an  annual  training  or 
qualification  requirement  that  is  satisfied  at  the 
operational  unit.  The  rationale  behind  this 
question  is  that  when  annual  testing  occurs  at  the 
unit,  there  is  a  strong  likelihood  that  NCOs  receive 
the  required  training  at  the  unit. 

8 

Universal 

task, 

uniform 

training 

required? 

Is  the  task  universal 
and  requiring 
uniformity  of  training 
across  the  MOS? 

When  the  task  is  relevant 
to  all  individuals  in  the 
MOS  and  also  requires 
that  all  Soldiers  be 
instructed  on  it  in  a 
standardized  manner, 
then  it  is  appropriate  to 
be  trained  at  the 
institution. 

This  question  has  two  parts.  Universality  is  about 
the  extent  to  which  the  task  is  relevant  to  NCOs 
across  the  Force  or  across  the  MOS,  regardless  of 
job  assignment.  Standardization  refers  to  how 
important  it  is  that  all  Soldiers  learn  to  conduct  the 
task  using  the  same  procedure.  This  question 
asks  whether  most  NCOs  will  employ  the  task  on 
the  job,  and  whether  they  must  know  or  use  the 
book  standard  to  be  effective. 

9 

Reliable  and 
controlled 
assessment 
? 

Is  a  reliable  and 
controlled  assessment 
highly  important? 

Tasks  that  require  extra 
assurance  that  Soldiers 
have  grasped  the 
important  concepts  and 
performance  steps  are 
better  suited  for 
institutional  training, 
where  a  reliable 
assessment  is  more 
likely  to  always  take 
place. 

The  importance  associated  with  ensuring  task 
training  has  been  delivered  and  performance  has 
been  assessed  for  each  and  every  NCO.  Some 
tasks  require  an  extra  degree  of  assurance  that 
every  Soldier  who  has  gone  through  the  training 
actually  grasps  the  task  and  can  perform  it 
effectively.  This  question  refers  to  the  criticality 
that  an  instructor  be  present  to  ensure  each  and 
every  NCO  is  trained  to  standard. 
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Question  # 

Site  Selection  Tool  Questions 

Purpose 

Definition 

10 

Peer-to-peer 
learning 
critical  and 
required? 

Is  sharing  experiences 
with  peers  from  other 
units  critical  and 
required  to  maximize 
learning? 

When  trainees  gain  a 
significant  benefit  from 
hearing  about  peers' 
experiences  in  a  range  of 
other  mission  or  garrison 
contexts,  institutional 
training  is  often  more 
appropriate. 

The  degree  to  which  there  is  a  learning  advantage 
associated  with  exposure  to  the  experiences  of 
Soldiers  from  other  units  or  backgrounds.  When  a 
task,  such  as  a  combat  tactic,  must  be  adjusted 
for  execution  in  different  mission  types  or 
operational  environments,  there  is  great  learning 
value  in  hearing  examples  of  the  many  ways  to 
conduct  the  task.  This  question  refers  to  whether 
there  is  a  particular  benefit  from  exposure  to  peers 
from  different  units  who  can  speak  to  how  mission 
or  environmental  factors  impact  task  execution. 

11 

Criticality  to 
leadership? 

How  critical  is  the  task 
to  the  leadership  role? 

Tasks  with  a  heavy 
leadership  component 
may  be  most  amenable 
to  an  institutional  course. 

The  importance  or  centrality  of  the  task  for  being 
an  effective  leader.  This  question  is  about 
whether  or  not  the  task  is  one  of  the  core  functions 
of  a  leader. 

12 

Safety/equip 

ment 

available  at 
unit? 

Do  most  units  have 
the  safety  measures 
and  equipment  readily 
available  to  perform 
training? 

When  the  other  factors 
indicate  unit  training  is 
more  appropriate  AND 
the  unit  has  the  required 
equipment  and  safety 
measures  to  conduct  the 
training,  then  unit  training 
is  probably  optimal. 

Whether  the  necessary  equipment  or  safety 
measures  for  trainina  the  task  are  present  and 
available  at  most  operational  units.  This  question 
provides  a  check  to  ensure  that  if  a  task  is 
recommended  for  training  exclusively  at  the  unit, 
most  units  across  the  Army  will  have  the  means  to 
conduct  that  training. 
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